2021-02-11T05:58:27.842 INFO:root:teuthology version: 1.1.0-14fad317 2021-02-11T05:58:27.843 DEBUG:teuthology.run:Teuthology command: teuthology --description rgw/multisite/{clusters frontend/beast omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind} --verbose --owner scheduled_aemerson@teuthology --archive /home/teuthworker/archive/aemerson-2021-02-11_05:45:06-rgw-wip-easter-bunny-distro-basic-smithi/5874443 --name aemerson-2021-02-11_05:45:06-rgw-wip-easter-bunny-distro-basic-smithi -- /home/teuthworker/archive/aemerson-2021-02-11_05:45:06-rgw-wip-easter-bunny-distro-basic-smithi/5874443/orig.config.yaml 2021-02-11T05:58:40.677 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2021-02-11T05:58:40.738 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/aemerson-2021-02-11_05:45:06-rgw-wip-easter-bunny-distro-basic-smithi/5874443 branch: wip-easter-bunny description: rgw/multisite/{clusters frontend/beast omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind} email: aemerson@redhat.com first_in_suite: false job_id: '5874443' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: aemerson-2021-02-11_05:45:06-rgw-wip-easter-bunny-distro-basic-smithi nuke-on-error: true os_type: ubuntu overrides: admin_socket: branch: wip-easter-bunny ceph: conf: client: debug rgw: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= rgw curl low speed time: 300 rgw data log num shards: 4 rgw md log max shards: 4 rgw sync log trim interval: 0 rgw sync obj etag verify: true setgroup: ceph setuser: ceph global: osd heartbeat grace: 40 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon osd crush smoke test: false osd: debug filestore: 20 debug journal: 20 debug ms: 20 debug osd: 25 osd fast shutdown: false rocksdb delete range threshold: 0 osd.0: osd_max_omap_entries_per_request: 10 osd.1: osd_max_omap_entries_per_request: 1000 osd.2: osd_max_omap_entries_per_request: 10000 log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 84a8025e5a892a6879f75e17e03444bf443eae4b wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: 84a8025e5a892a6879f75e17e03444bf443eae4b rgw: compression type: random frontend: beast rgw-multisite: realm: is default: true name: test-realm zonegroups: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zonegroup zones: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zone1 - endpoints: - c2.client.0 is_default: true name: test-zone2 - endpoints: - c1.client.1 name: test-zone3 - endpoints: - c2.client.1 is_pubsub: true name: test-zone4 workunit: branch: wip-easter-bunny sha1: 84a8025e5a892a6879f75e17e03444bf443eae4b owner: scheduled_aemerson@teuthology priority: 70 repo: git://git.ceph.com/ceph-ci.git roles: - - c1.mon.a - c1.mgr.x - c1.osd.0 - c1.osd.1 - c1.osd.2 - c1.client.0 - c1.client.1 - - c2.mon.a - c2.mgr.x - c2.osd.0 - c2.osd.1 - c2.osd.2 - c2.client.0 - c2.client.1 sha1: 84a8025e5a892a6879f75e17e03444bf443eae4b sleep_before_teardown: 0 suite: rgw suite_branch: wip-easter-bunny suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_84a8025e5a892a6879f75e17e03444bf443eae4b/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: 84a8025e5a892a6879f75e17e03444bf443eae4b targets: smithi061.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9nFZMnqNQMV9nVvBUk/hdcUhFPn2XCz9ZkCiQbtge7HeikSAfUCyOCHSfz0S1pmOcIbc6LkEm4xWkAeLB9/pdSZ+PzVLBUanYR6RkuRKO4xTYn+JavSZmJCi3NQ5NcsBR04RRvp/saikmxQmmdlwsgajmkmavyzkL90CEquTby7jL4PERDRhH/qpnwlzPWgaaiea3k9Yh41Jvtd1xQi0+1IsNGsmoVgvfz90gsgnXHmuuYXUTC9xbXtlZnq1pOmBAtWlaf0n/YpZUMki+jsM6TOwG3OOGZ2m69P7yNNbjhZiBmDSi0lBcx7cMG7lyYWJrFO+6a+MvQA1L/Z1hR14X smithi172.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKc1kIvUvofa8RKQkSgMIYhvCE2w7RZG4ZgcVx5kPL/pTrHdUfID6cH6NblNsgR97jYHYb7olU29Qx8ZIgSSI3zbdAxGNt1IgvEfG58vL1U2g0Ln3iIV/ZRIzyl22IbRjb4UNstYPCxc5lX9Ni+ngwwE7XmvflhXJAL5Ykpq74qpcvhcG0T/JCADGX2q+LhAUSmgb6vYzWdwz1GhObmV2W80cMQBpmsoffW5w3w7uu6rHdNA0rK1C9x0/bqDMp1TUViv3pe7jxD3JlufcDDIszdtwc2s4n1w0HhzbQmV1tJZUD1tKjtTOoqWuFdMb9t//7ZKqjatFUjJzKc3ZWv86X tasks: - install: null - ceph: cluster: c1 - ceph: cluster: c2 - rgw: c1.client.0: port: 8000 c1.client.1: port: 8001 c2.client.0: port: 8000 c2.client.1: port: 8001 - rgw-multisite: null - rgw-multisite-tests: config: reconfigure_delay: 60 teuthology_branch: master timestamp: 2021-02-11_05:45:06 tube: smithi user: aemerson verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.29888 2021-02-11T05:58:40.738 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_84a8025e5a892a6879f75e17e03444bf443eae4b/qa; will attempt to use it 2021-02-11T05:58:40.749 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_84a8025e5a892a6879f75e17e03444bf443eae4b/qa/tasks 2021-02-11T05:58:40.749 INFO:teuthology.run_tasks:Running task internal.check_packages... 2021-02-11T05:58:40.750 INFO:teuthology.task.internal:Checking packages... 2021-02-11T05:58:40.799 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'basic' and ceph hash '84a8025e5a892a6879f75e17e03444bf443eae4b' 2021-02-11T05:58:40.799 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2021-02-11T05:58:40.799 INFO:teuthology.packaging:ref: None 2021-02-11T05:58:40.799 INFO:teuthology.packaging:tag: None 2021-02-11T05:58:40.799 INFO:teuthology.packaging:branch: wip-easter-bunny 2021-02-11T05:58:40.799 INFO:teuthology.packaging:sha1: 84a8025e5a892a6879f75e17e03444bf443eae4b 2021-02-11T05:58:40.799 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&ref=wip-easter-bunny 2021-02-11T05:58:40.970 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-711-g84a8025e-1bionic 2021-02-11T05:58:40.971 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2021-02-11T05:58:40.978 INFO:teuthology.task.internal:no buildpackages task found 2021-02-11T05:58:40.978 INFO:teuthology.run_tasks:Running task internal.save_config... 2021-02-11T05:58:41.650 INFO:teuthology.task.internal:Saving configuration 2021-02-11T05:58:41.665 INFO:teuthology.run_tasks:Running task internal.check_lock... 2021-02-11T05:58:42.721 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-11T05:58:42.743 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-02-11 05:52:42.478147', 'description': '/home/teuthworker/archive/aemerson-2021-02-11_05:45:06-rgw-wip-easter-bunny-distro-basic-smithi/5874443', 'locked_by': 'scheduled_aemerson@teuthology', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAII1rjTv5z+9NR0Tc4ckdzHaxusBjpCJcveVHH5NLnA8E', 'os_version': '18.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi061.front.sepia.ceph.com'} 2021-02-11T05:58:42.761 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-02-11 05:52:42.478176', 'description': '/home/teuthworker/archive/aemerson-2021-02-11_05:45:06-rgw-wip-easter-bunny-distro-basic-smithi/5874443', 'locked_by': 'scheduled_aemerson@teuthology', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINi7aWImY/1Y3kTNnXRRTVZb1ZUM+GT7H+Z/IOLmCQTn', 'os_version': '18.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi172.front.sepia.ceph.com'} 2021-02-11T05:59:02.249 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2021-02-11T05:59:02.257 INFO:teuthology.task.internal:roles: ubuntu@smithi061.front.sepia.ceph.com - ['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'] 2021-02-11T05:59:02.258 INFO:teuthology.task.internal:roles: ubuntu@smithi172.front.sepia.ceph.com - ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1'] 2021-02-11T05:59:02.258 INFO:teuthology.run_tasks:Running task console_log... 2021-02-11T05:59:15.860 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fa6c164e378>, signals=[15]) 2021-02-11T05:59:15.861 INFO:teuthology.run_tasks:Running task internal.connect... 2021-02-11T05:59:15.869 INFO:teuthology.task.internal:Opening connections... 2021-02-11T05:59:15.869 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi061.front.sepia.ceph.com 2021-02-11T05:59:15.872 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-02-11T05:59:15.949 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi172.front.sepia.ceph.com 2021-02-11T05:59:15.950 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-02-11T05:59:16.024 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2021-02-11T05:59:16.640 DEBUG:teuthology.orchestra.run.smithi061:> uname -m 2021-02-11T05:59:16.646 INFO:teuthology.orchestra.run.smithi061.stdout:x86_64 2021-02-11T05:59:16.647 DEBUG:teuthology.orchestra.run.smithi061:> cat /etc/os-release 2021-02-11T05:59:16.693 INFO:teuthology.orchestra.run.smithi061.stdout:NAME="Ubuntu" 2021-02-11T05:59:16.693 INFO:teuthology.orchestra.run.smithi061.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2021-02-11T05:59:16.693 INFO:teuthology.orchestra.run.smithi061.stdout:ID=ubuntu 2021-02-11T05:59:16.693 INFO:teuthology.orchestra.run.smithi061.stdout:ID_LIKE=debian 2021-02-11T05:59:16.693 INFO:teuthology.orchestra.run.smithi061.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2021-02-11T05:59:16.693 INFO:teuthology.orchestra.run.smithi061.stdout:VERSION_ID="18.04" 2021-02-11T05:59:16.693 INFO:teuthology.orchestra.run.smithi061.stdout:HOME_URL="https://www.ubuntu.com/" 2021-02-11T05:59:16.693 INFO:teuthology.orchestra.run.smithi061.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-02-11T05:59:16.693 INFO:teuthology.orchestra.run.smithi061.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-02-11T05:59:16.693 INFO:teuthology.orchestra.run.smithi061.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-02-11T05:59:16.694 INFO:teuthology.orchestra.run.smithi061.stdout:VERSION_CODENAME=bionic 2021-02-11T05:59:16.694 INFO:teuthology.orchestra.run.smithi061.stdout:UBUNTU_CODENAME=bionic 2021-02-11T05:59:16.694 INFO:teuthology.lock.ops:Updating smithi061.front.sepia.ceph.com on lock server 2021-02-11T05:59:16.711 DEBUG:teuthology.orchestra.run.smithi172:> uname -m 2021-02-11T05:59:16.716 INFO:teuthology.orchestra.run.smithi172.stdout:x86_64 2021-02-11T05:59:16.717 DEBUG:teuthology.orchestra.run.smithi172:> cat /etc/os-release 2021-02-11T05:59:16.762 INFO:teuthology.orchestra.run.smithi172.stdout:NAME="Ubuntu" 2021-02-11T05:59:16.763 INFO:teuthology.orchestra.run.smithi172.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2021-02-11T05:59:16.763 INFO:teuthology.orchestra.run.smithi172.stdout:ID=ubuntu 2021-02-11T05:59:16.763 INFO:teuthology.orchestra.run.smithi172.stdout:ID_LIKE=debian 2021-02-11T05:59:16.763 INFO:teuthology.orchestra.run.smithi172.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2021-02-11T05:59:16.763 INFO:teuthology.orchestra.run.smithi172.stdout:VERSION_ID="18.04" 2021-02-11T05:59:16.763 INFO:teuthology.orchestra.run.smithi172.stdout:HOME_URL="https://www.ubuntu.com/" 2021-02-11T05:59:16.763 INFO:teuthology.orchestra.run.smithi172.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-02-11T05:59:16.763 INFO:teuthology.orchestra.run.smithi172.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-02-11T05:59:16.763 INFO:teuthology.orchestra.run.smithi172.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-02-11T05:59:16.763 INFO:teuthology.orchestra.run.smithi172.stdout:VERSION_CODENAME=bionic 2021-02-11T05:59:16.763 INFO:teuthology.orchestra.run.smithi172.stdout:UBUNTU_CODENAME=bionic 2021-02-11T05:59:16.764 INFO:teuthology.lock.ops:Updating smithi172.front.sepia.ceph.com on lock server 2021-02-11T05:59:16.778 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2021-02-11T05:59:17.282 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2021-02-11T05:59:18.851 INFO:teuthology.task.internal:Checking for old test directory... 2021-02-11T05:59:19.779 DEBUG:teuthology.orchestra.run.smithi061:> test '!' -e /home/ubuntu/cephtest 2021-02-11T05:59:19.781 DEBUG:teuthology.orchestra.run.smithi172:> test '!' -e /home/ubuntu/cephtest 2021-02-11T05:59:19.786 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2021-02-11T05:59:19.793 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2021-02-11T05:59:19.794 DEBUG:teuthology.orchestra.run.smithi061:> test -z $(ls -A /var/lib/ceph) 2021-02-11T05:59:19.829 DEBUG:teuthology.orchestra.run.smithi172:> test -z $(ls -A /var/lib/ceph) 2021-02-11T05:59:19.851 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2021-02-11T05:59:22.175 INFO:teuthology.run_tasks:Running task kernel... 2021-02-11T05:59:25.381 DEBUG:teuthology.task.kernel:normalized config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c1.mgr.x': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.0': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.1': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.2': {'kdb': True, 'sha1': 'distro'}, 'c1.client.0': {'kdb': True, 'sha1': 'distro'}, 'c1.client.1': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mgr.x': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.0': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.1': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.2': {'kdb': True, 'sha1': 'distro'}, 'c2.client.0': {'kdb': True, 'sha1': 'distro'}, 'c2.client.1': {'kdb': True, 'sha1': 'distro'}} 2021-02-11T05:59:25.382 INFO:teuthology.task.kernel:config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2021-02-11T05:59:25.382 DEBUG:teuthology.orchestra.run.smithi061:> uname -r 2021-02-11T05:59:25.388 INFO:teuthology.orchestra.run.smithi061.stdout:4.15.0-134-generic 2021-02-11T05:59:25.388 INFO:teuthology.task.kernel:Running kernel on smithi061: 4.15.0-134-generic 2021-02-11T05:59:25.388 DEBUG:teuthology.orchestra.run.smithi061:> sudo apt-get clean 2021-02-11T05:59:25.499 DEBUG:teuthology.orchestra.run.smithi061:> sudo apt-get update 2021-02-11T05:59:25.663 INFO:teuthology.orchestra.run.smithi061.stdout:Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2021-02-11T05:59:25.780 INFO:teuthology.orchestra.run.smithi061.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2021-02-11T05:59:25.860 INFO:teuthology.orchestra.run.smithi061.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2021-02-11T05:59:25.943 INFO:teuthology.orchestra.run.smithi061.stdout:Get:4 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [1,543 kB] 2021-02-11T05:59:26.078 INFO:teuthology.orchestra.run.smithi061.stdout:Get:5 http://security.ubuntu.com/ubuntu bionic-security/main i386 Packages [915 kB] 2021-02-11T05:59:26.093 INFO:teuthology.orchestra.run.smithi061.stdout:Get:6 http://security.ubuntu.com/ubuntu bionic-security/main Translation-en [298 kB] 2021-02-11T05:59:26.096 INFO:teuthology.orchestra.run.smithi061.stdout:Get:7 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [226 kB] 2021-02-11T05:59:26.098 INFO:teuthology.orchestra.run.smithi061.stdout:Get:8 http://security.ubuntu.com/ubuntu bionic-security/restricted Translation-en [29.9 kB] 2021-02-11T05:59:26.099 INFO:teuthology.orchestra.run.smithi061.stdout:Get:9 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1,109 kB] 2021-02-11T05:59:26.111 INFO:teuthology.orchestra.run.smithi061.stdout:Get:10 http://security.ubuntu.com/ubuntu bionic-security/universe i386 Packages [975 kB] 2021-02-11T05:59:26.117 INFO:teuthology.orchestra.run.smithi061.stdout:Get:11 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2021-02-11T05:59:26.123 INFO:teuthology.orchestra.run.smithi061.stdout:Get:12 http://security.ubuntu.com/ubuntu bionic-security/universe Translation-en [248 kB] 2021-02-11T05:59:26.259 INFO:teuthology.orchestra.run.smithi061.stdout:Get:13 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [1,883 kB] 2021-02-11T05:59:26.652 INFO:teuthology.orchestra.run.smithi061.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages [1,218 kB] 2021-02-11T05:59:26.687 INFO:teuthology.orchestra.run.smithi061.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic-updates/main Translation-en [390 kB] 2021-02-11T05:59:26.698 INFO:teuthology.orchestra.run.smithi061.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [247 kB] 2021-02-11T05:59:26.704 INFO:teuthology.orchestra.run.smithi061.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic-updates/restricted Translation-en [33.4 kB] 2021-02-11T05:59:26.705 INFO:teuthology.orchestra.run.smithi061.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic-updates/universe i386 Packages [1,560 kB] 2021-02-11T05:59:26.773 INFO:teuthology.orchestra.run.smithi061.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1,718 kB] 2021-02-11T05:59:26.823 INFO:teuthology.orchestra.run.smithi061.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic-updates/universe Translation-en [363 kB] 2021-02-11T05:59:26.829 INFO:teuthology.orchestra.run.smithi061.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [31.8 kB] 2021-02-11T05:59:26.829 INFO:teuthology.orchestra.run.smithi061.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse i386 Packages [14.3 kB] 2021-02-11T05:59:26.830 INFO:teuthology.orchestra.run.smithi061.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse Translation-en [7,300 B] 2021-02-11T05:59:27.975 INFO:teuthology.orchestra.run.smithi061.stdout:Fetched 13.1 MB in 2s (5,867 kB/s) 2021-02-11T05:59:38.461 INFO:teuthology.orchestra.run.smithi061.stdout:Reading package lists... 2021-02-11T05:59:38.462 DEBUG:teuthology.orchestra.run.smithi061:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2021-02-11T05:59:38.548 INFO:teuthology.orchestra.run.smithi061.stdout:Reading package lists... 2021-02-11T05:59:38.812 INFO:teuthology.orchestra.run.smithi061.stdout:Building dependency tree... 2021-02-11T05:59:38.813 INFO:teuthology.orchestra.run.smithi061.stdout:Reading state information... 2021-02-11T05:59:38.897 INFO:teuthology.orchestra.run.smithi061.stderr:E: Unable to locate package linux-image-current-generic 2021-02-11T05:59:38.898 DEBUG:teuthology.orchestra.run:got remote process result: 100 2021-02-11T05:59:38.899 DEBUG:teuthology.orchestra.run.smithi061:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2021-02-11T05:59:38.981 INFO:teuthology.orchestra.run.smithi061.stdout:Reading package lists... 2021-02-11T05:59:39.230 INFO:teuthology.orchestra.run.smithi061.stdout:Building dependency tree... 2021-02-11T05:59:39.230 INFO:teuthology.orchestra.run.smithi061.stdout:Reading state information... 2021-02-11T05:59:39.429 INFO:teuthology.orchestra.run.smithi061.stdout:The following additional packages will be installed: 2021-02-11T05:59:39.429 INFO:teuthology.orchestra.run.smithi061.stdout: linux-generic linux-headers-4.15.0-135 linux-headers-4.15.0-135-generic 2021-02-11T05:59:39.429 INFO:teuthology.orchestra.run.smithi061.stdout: linux-headers-generic linux-image-4.15.0-135-generic 2021-02-11T05:59:39.430 INFO:teuthology.orchestra.run.smithi061.stdout: linux-modules-4.15.0-135-generic linux-modules-extra-4.15.0-135-generic 2021-02-11T05:59:39.432 INFO:teuthology.orchestra.run.smithi061.stdout:Suggested packages: 2021-02-11T05:59:39.432 INFO:teuthology.orchestra.run.smithi061.stdout: fdutils linux-doc-4.15.0 | linux-source-4.15.0 linux-tools 2021-02-11T05:59:39.432 INFO:teuthology.orchestra.run.smithi061.stdout:Recommended packages: 2021-02-11T05:59:39.432 INFO:teuthology.orchestra.run.smithi061.stdout: thermald 2021-02-11T05:59:39.468 INFO:teuthology.orchestra.run.smithi061.stdout:The following NEW packages will be installed: 2021-02-11T05:59:39.468 INFO:teuthology.orchestra.run.smithi061.stdout: linux-headers-4.15.0-135 linux-headers-4.15.0-135-generic 2021-02-11T05:59:39.468 INFO:teuthology.orchestra.run.smithi061.stdout: linux-image-4.15.0-135-generic linux-modules-4.15.0-135-generic 2021-02-11T05:59:39.468 INFO:teuthology.orchestra.run.smithi061.stdout: linux-modules-extra-4.15.0-135-generic 2021-02-11T05:59:39.470 INFO:teuthology.orchestra.run.smithi061.stdout:The following packages will be upgraded: 2021-02-11T05:59:39.471 INFO:teuthology.orchestra.run.smithi061.stdout: linux-generic linux-headers-generic linux-image-generic 2021-02-11T05:59:39.645 INFO:teuthology.orchestra.run.smithi061.stdout:3 upgraded, 5 newly installed, 0 to remove and 23 not upgraded. 2021-02-11T05:59:39.646 INFO:teuthology.orchestra.run.smithi061.stdout:Need to get 66.0 MB of archives. 2021-02-11T05:59:39.646 INFO:teuthology.orchestra.run.smithi061.stdout:After this operation, 334 MB of additional disk space will be used. 2021-02-11T05:59:39.646 INFO:teuthology.orchestra.run.smithi061.stdout:Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-4.15.0-135-generic amd64 4.15.0-135.139 [13.1 MB] 2021-02-11T05:59:40.622 INFO:teuthology.orchestra.run.smithi061.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-4.15.0-135-generic amd64 4.15.0-135.139 [8,019 kB] 2021-02-11T05:59:40.872 INFO:teuthology.orchestra.run.smithi061.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-extra-4.15.0-135-generic amd64 4.15.0-135.139 [32.8 MB] 2021-02-11T05:59:41.802 INFO:teuthology.orchestra.run.smithi061.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-generic amd64 4.15.0.135.122 [1,868 B] 2021-02-11T05:59:41.802 INFO:teuthology.orchestra.run.smithi061.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-generic amd64 4.15.0.135.122 [2,592 B] 2021-02-11T05:59:41.802 INFO:teuthology.orchestra.run.smithi061.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-135 all 4.15.0-135.139 [10.9 MB] 2021-02-11T05:59:42.114 INFO:teuthology.orchestra.run.smithi061.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-135-generic amd64 4.15.0-135.139 [1,097 kB] 2021-02-11T05:59:42.131 INFO:teuthology.orchestra.run.smithi061.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-generic amd64 4.15.0.135.122 [2,488 B] 2021-02-11T05:59:42.638 INFO:teuthology.orchestra.run.smithi061.stdout:Fetched 66.0 MB in 3s (24.9 MB/s) 2021-02-11T05:59:46.780 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package linux-modules-4.15.0-135-generic. 2021-02-11T05:59:46.780 INFO:teuthology.orchestra.run.smithi061.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102079 files and directories currently installed.) 2021-02-11T05:59:46.780 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../0-linux-modules-4.15.0-135-generic_4.15.0-135.139_amd64.deb ... 2021-02-11T05:59:46.780 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking linux-modules-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-11T05:59:46.780 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package linux-image-4.15.0-135-generic. 2021-02-11T05:59:46.780 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../1-linux-image-4.15.0-135-generic_4.15.0-135.139_amd64.deb ... 2021-02-11T05:59:46.780 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking linux-image-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-11T05:59:46.780 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package linux-modules-extra-4.15.0-135-generic. 2021-02-11T05:59:46.780 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../2-linux-modules-extra-4.15.0-135-generic_4.15.0-135.139_amd64.deb ... 2021-02-11T05:59:46.781 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking linux-modules-extra-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-11T05:59:48.948 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../3-linux-generic_4.15.0.135.122_amd64.deb ... 2021-02-11T06:00:00.423 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking linux-generic (4.15.0.135.122) over (4.15.0.134.121) ... 2021-02-11T06:00:00.424 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../4-linux-image-generic_4.15.0.135.122_amd64.deb ... 2021-02-11T06:00:00.424 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking linux-image-generic (4.15.0.135.122) over (4.15.0.134.121) ... 2021-02-11T06:00:00.424 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package linux-headers-4.15.0-135. 2021-02-11T06:00:00.424 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../5-linux-headers-4.15.0-135_4.15.0-135.139_all.deb ... 2021-02-11T06:00:00.424 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking linux-headers-4.15.0-135 (4.15.0-135.139) ... 2021-02-11T06:00:00.424 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package linux-headers-4.15.0-135-generic. 2021-02-11T06:00:00.424 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../6-linux-headers-4.15.0-135-generic_4.15.0-135.139_amd64.deb ... 2021-02-11T06:00:00.424 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking linux-headers-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-11T06:00:00.424 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../7-linux-headers-generic_4.15.0.135.122_amd64.deb ... 2021-02-11T06:00:00.424 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking linux-headers-generic (4.15.0.135.122) over (4.15.0.134.121) ... 2021-02-11T06:00:00.424 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up linux-modules-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-11T06:00:00.425 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up linux-headers-4.15.0-135 (4.15.0-135.139) ... 2021-02-11T06:00:00.425 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up linux-image-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-11T06:00:00.425 INFO:teuthology.orchestra.run.smithi061.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-4.15.0-135-generic 2021-02-11T06:00:00.425 INFO:teuthology.orchestra.run.smithi061.stdout:I: /initrd.img is now a symlink to boot/initrd.img-4.15.0-135-generic 2021-02-11T06:00:00.425 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up linux-modules-extra-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-11T06:00:00.425 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up linux-headers-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-11T06:00:00.425 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up linux-image-generic (4.15.0.135.122) ... 2021-02-11T06:00:00.425 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up linux-headers-generic (4.15.0.135.122) ... 2021-02-11T06:00:00.425 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up linux-generic (4.15.0.135.122) ... 2021-02-11T06:00:00.444 INFO:teuthology.orchestra.run.smithi061.stdout:Processing triggers for linux-image-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-11T06:00:01.475 INFO:teuthology.orchestra.run.smithi061.stdout:/etc/kernel/postinst.d/initramfs-tools: 2021-02-11T06:00:01.777 INFO:teuthology.orchestra.run.smithi061.stdout:update-initramfs: Generating /boot/initrd.img-4.15.0-135-generic 2021-02-11T06:00:16.291 INFO:teuthology.orchestra.run.smithi061.stdout:/etc/kernel/postinst.d/zz-update-grub: 2021-02-11T06:00:16.292 INFO:teuthology.orchestra.run.smithi061.stdout:Sourcing file `/etc/default/grub' 2021-02-11T06:00:16.336 INFO:teuthology.orchestra.run.smithi061.stdout:Generating grub configuration file ... 2021-02-11T06:00:16.726 INFO:teuthology.orchestra.run.smithi061.stdout:Found linux image: /boot/vmlinuz-4.15.0-135-generic 2021-02-11T06:00:16.753 INFO:teuthology.orchestra.run.smithi061.stdout:Found initrd image: /boot/initrd.img-4.15.0-135-generic 2021-02-11T06:00:17.132 INFO:teuthology.orchestra.run.smithi061.stdout:Found linux image: /boot/vmlinuz-4.15.0-134-generic 2021-02-11T06:00:17.162 INFO:teuthology.orchestra.run.smithi061.stdout:Found initrd image: /boot/initrd.img-4.15.0-134-generic 2021-02-11T06:00:18.043 INFO:teuthology.orchestra.run.smithi061.stdout:done 2021-02-11T06:00:19.644 DEBUG:teuthology.orchestra.run.smithi061:> dpkg -s linux-image-generic 2021-02-11T06:00:19.670 INFO:teuthology.orchestra.run.smithi061.stdout:Package: linux-image-generic 2021-02-11T06:00:19.670 INFO:teuthology.orchestra.run.smithi061.stdout:Status: install ok installed 2021-02-11T06:00:19.670 INFO:teuthology.orchestra.run.smithi061.stdout:Priority: optional 2021-02-11T06:00:19.670 INFO:teuthology.orchestra.run.smithi061.stdout:Section: kernel 2021-02-11T06:00:19.670 INFO:teuthology.orchestra.run.smithi061.stdout:Installed-Size: 17 2021-02-11T06:00:19.670 INFO:teuthology.orchestra.run.smithi061.stdout:Maintainer: Ubuntu Kernel Team 2021-02-11T06:00:19.670 INFO:teuthology.orchestra.run.smithi061.stdout:Architecture: amd64 2021-02-11T06:00:19.670 INFO:teuthology.orchestra.run.smithi061.stdout:Source: linux-meta 2021-02-11T06:00:19.670 INFO:teuthology.orchestra.run.smithi061.stdout:Version: 4.15.0.135.122 2021-02-11T06:00:19.671 INFO:teuthology.orchestra.run.smithi061.stdout:Provides: wireguard-modules (= 1.0.20201112-1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.10) 2021-02-11T06:00:19.671 INFO:teuthology.orchestra.run.smithi061.stdout:Depends: linux-image-4.15.0-135-generic, linux-modules-extra-4.15.0-135-generic, linux-firmware, intel-microcode, amd64-microcode 2021-02-11T06:00:19.671 INFO:teuthology.orchestra.run.smithi061.stdout:Recommends: thermald 2021-02-11T06:00:19.671 INFO:teuthology.orchestra.run.smithi061.stdout:Description: Generic Linux kernel image 2021-02-11T06:00:19.671 INFO:teuthology.orchestra.run.smithi061.stdout: This package will always depend on the latest generic kernel image 2021-02-11T06:00:19.671 INFO:teuthology.orchestra.run.smithi061.stdout: available. 2021-02-11T06:00:19.671 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.15.0-134-generic Expected: 4.15.0-135-generic 2021-02-11T06:00:19.671 DEBUG:teuthology.orchestra.run.smithi172:> uname -r 2021-02-11T06:00:19.677 INFO:teuthology.orchestra.run.smithi172.stdout:4.15.0-134-generic 2021-02-11T06:00:19.678 INFO:teuthology.task.kernel:Running kernel on smithi172: 4.15.0-134-generic 2021-02-11T06:00:19.678 DEBUG:teuthology.orchestra.run.smithi172:> sudo apt-get clean 2021-02-11T06:00:19.794 DEBUG:teuthology.orchestra.run.smithi172:> sudo apt-get update 2021-02-11T06:00:20.031 INFO:teuthology.orchestra.run.smithi172.stdout:Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2021-02-11T06:00:20.053 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2021-02-11T06:00:20.110 INFO:teuthology.orchestra.run.smithi172.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2021-02-11T06:00:20.394 INFO:teuthology.orchestra.run.smithi172.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2021-02-11T06:00:20.441 INFO:teuthology.orchestra.run.smithi172.stdout:Get:5 http://security.ubuntu.com/ubuntu bionic-security/main i386 Packages [915 kB] 2021-02-11T06:00:20.574 INFO:teuthology.orchestra.run.smithi172.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [1,883 kB] 2021-02-11T06:00:20.825 INFO:teuthology.orchestra.run.smithi172.stdout:Get:7 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [1,543 kB] 2021-02-11T06:00:20.936 INFO:teuthology.orchestra.run.smithi172.stdout:Get:8 http://security.ubuntu.com/ubuntu bionic-security/main Translation-en [298 kB] 2021-02-11T06:00:20.953 INFO:teuthology.orchestra.run.smithi172.stdout:Get:9 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [226 kB] 2021-02-11T06:00:20.964 INFO:teuthology.orchestra.run.smithi172.stdout:Get:10 http://security.ubuntu.com/ubuntu bionic-security/restricted Translation-en [29.9 kB] 2021-02-11T06:00:20.965 INFO:teuthology.orchestra.run.smithi172.stdout:Get:11 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1,109 kB] 2021-02-11T06:00:20.978 INFO:teuthology.orchestra.run.smithi172.stdout:Get:12 http://archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages [1,218 kB] 2021-02-11T06:00:20.998 INFO:teuthology.orchestra.run.smithi172.stdout:Get:13 http://security.ubuntu.com/ubuntu bionic-security/universe i386 Packages [975 kB] 2021-02-11T06:00:21.002 INFO:teuthology.orchestra.run.smithi172.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic-updates/main Translation-en [390 kB] 2021-02-11T06:00:21.008 INFO:teuthology.orchestra.run.smithi172.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [247 kB] 2021-02-11T06:00:21.011 INFO:teuthology.orchestra.run.smithi172.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic-updates/restricted Translation-en [33.4 kB] 2021-02-11T06:00:21.012 INFO:teuthology.orchestra.run.smithi172.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1,718 kB] 2021-02-11T06:00:21.027 INFO:teuthology.orchestra.run.smithi172.stdout:Get:18 http://security.ubuntu.com/ubuntu bionic-security/universe Translation-en [248 kB] 2021-02-11T06:00:21.085 INFO:teuthology.orchestra.run.smithi172.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic-updates/universe i386 Packages [1,560 kB] 2021-02-11T06:00:21.150 INFO:teuthology.orchestra.run.smithi172.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic-updates/universe Translation-en [363 kB] 2021-02-11T06:00:21.153 INFO:teuthology.orchestra.run.smithi172.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse i386 Packages [14.3 kB] 2021-02-11T06:00:21.154 INFO:teuthology.orchestra.run.smithi172.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [31.8 kB] 2021-02-11T06:00:21.154 INFO:teuthology.orchestra.run.smithi172.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse Translation-en [7,300 B] 2021-02-11T06:00:22.582 INFO:teuthology.orchestra.run.smithi172.stdout:Fetched 13.1 MB in 3s (5,108 kB/s) 2021-02-11T06:00:23.980 INFO:teuthology.orchestra.run.smithi172.stdout:Reading package lists... 2021-02-11T06:00:40.809 DEBUG:teuthology.orchestra.run.smithi172:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2021-02-11T06:00:40.895 INFO:teuthology.orchestra.run.smithi172.stdout:Reading package lists... 2021-02-11T06:00:41.157 INFO:teuthology.orchestra.run.smithi172.stdout:Building dependency tree... 2021-02-11T06:00:41.158 INFO:teuthology.orchestra.run.smithi172.stdout:Reading state information... 2021-02-11T06:00:41.239 INFO:teuthology.orchestra.run.smithi172.stderr:E: Unable to locate package linux-image-current-generic 2021-02-11T06:00:41.240 DEBUG:teuthology.orchestra.run:got remote process result: 100 2021-02-11T06:00:41.241 DEBUG:teuthology.orchestra.run.smithi172:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2021-02-11T06:00:41.323 INFO:teuthology.orchestra.run.smithi172.stdout:Reading package lists... 2021-02-11T06:00:41.580 INFO:teuthology.orchestra.run.smithi172.stdout:Building dependency tree... 2021-02-11T06:00:41.580 INFO:teuthology.orchestra.run.smithi172.stdout:Reading state information... 2021-02-11T06:00:41.755 INFO:teuthology.orchestra.run.smithi172.stdout:The following additional packages will be installed: 2021-02-11T06:00:41.755 INFO:teuthology.orchestra.run.smithi172.stdout: linux-generic linux-headers-4.15.0-135 linux-headers-4.15.0-135-generic 2021-02-11T06:00:41.755 INFO:teuthology.orchestra.run.smithi172.stdout: linux-headers-generic linux-image-4.15.0-135-generic 2021-02-11T06:00:41.755 INFO:teuthology.orchestra.run.smithi172.stdout: linux-modules-4.15.0-135-generic linux-modules-extra-4.15.0-135-generic 2021-02-11T06:00:41.757 INFO:teuthology.orchestra.run.smithi172.stdout:Suggested packages: 2021-02-11T06:00:41.757 INFO:teuthology.orchestra.run.smithi172.stdout: fdutils linux-doc-4.15.0 | linux-source-4.15.0 linux-tools 2021-02-11T06:00:41.757 INFO:teuthology.orchestra.run.smithi172.stdout:Recommended packages: 2021-02-11T06:00:41.757 INFO:teuthology.orchestra.run.smithi172.stdout: thermald 2021-02-11T06:00:41.788 INFO:teuthology.orchestra.run.smithi172.stdout:The following NEW packages will be installed: 2021-02-11T06:00:41.788 INFO:teuthology.orchestra.run.smithi172.stdout: linux-headers-4.15.0-135 linux-headers-4.15.0-135-generic 2021-02-11T06:00:41.788 INFO:teuthology.orchestra.run.smithi172.stdout: linux-image-4.15.0-135-generic linux-modules-4.15.0-135-generic 2021-02-11T06:00:41.789 INFO:teuthology.orchestra.run.smithi172.stdout: linux-modules-extra-4.15.0-135-generic 2021-02-11T06:00:41.790 INFO:teuthology.orchestra.run.smithi172.stdout:The following packages will be upgraded: 2021-02-11T06:00:41.791 INFO:teuthology.orchestra.run.smithi172.stdout: linux-generic linux-headers-generic linux-image-generic 2021-02-11T06:00:41.988 INFO:teuthology.orchestra.run.smithi172.stdout:3 upgraded, 5 newly installed, 0 to remove and 23 not upgraded. 2021-02-11T06:00:41.988 INFO:teuthology.orchestra.run.smithi172.stdout:Need to get 66.0 MB of archives. 2021-02-11T06:00:41.988 INFO:teuthology.orchestra.run.smithi172.stdout:After this operation, 334 MB of additional disk space will be used. 2021-02-11T06:00:41.988 INFO:teuthology.orchestra.run.smithi172.stdout:Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-4.15.0-135-generic amd64 4.15.0-135.139 [13.1 MB] 2021-02-11T06:00:42.985 INFO:teuthology.orchestra.run.smithi172.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-4.15.0-135-generic amd64 4.15.0-135.139 [8,019 kB] 2021-02-11T06:00:43.184 INFO:teuthology.orchestra.run.smithi172.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-extra-4.15.0-135-generic amd64 4.15.0-135.139 [32.8 MB] 2021-02-11T06:00:44.159 INFO:teuthology.orchestra.run.smithi172.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-generic amd64 4.15.0.135.122 [1,868 B] 2021-02-11T06:00:44.160 INFO:teuthology.orchestra.run.smithi172.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-generic amd64 4.15.0.135.122 [2,592 B] 2021-02-11T06:00:44.160 INFO:teuthology.orchestra.run.smithi172.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-135 all 4.15.0-135.139 [10.9 MB] 2021-02-11T06:00:44.473 INFO:teuthology.orchestra.run.smithi172.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-135-generic amd64 4.15.0-135.139 [1,097 kB] 2021-02-11T06:00:45.389 INFO:teuthology.orchestra.run.smithi172.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-generic amd64 4.15.0.135.122 [2,488 B] 2021-02-11T06:00:45.389 INFO:teuthology.orchestra.run.smithi172.stdout:Fetched 66.0 MB in 3s (24.5 MB/s) 2021-02-11T06:00:45.389 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package linux-modules-4.15.0-135-generic. 2021-02-11T06:00:45.389 INFO:teuthology.orchestra.run.smithi172.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102079 files and directories currently installed.) 2021-02-11T06:00:45.389 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../0-linux-modules-4.15.0-135-generic_4.15.0-135.139_amd64.deb ... 2021-02-11T06:00:45.389 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking linux-modules-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-11T06:00:46.945 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package linux-image-4.15.0-135-generic. 2021-02-11T06:00:46.958 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../1-linux-image-4.15.0-135-generic_4.15.0-135.139_amd64.deb ... 2021-02-11T06:00:47.058 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking linux-image-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-11T06:00:47.539 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package linux-modules-extra-4.15.0-135-generic. 2021-02-11T06:01:06.350 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../2-linux-modules-extra-4.15.0-135-generic_4.15.0-135.139_amd64.deb ... 2021-02-11T06:01:06.351 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking linux-modules-extra-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-11T06:01:06.351 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../3-linux-generic_4.15.0.135.122_amd64.deb ... 2021-02-11T06:01:06.351 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking linux-generic (4.15.0.135.122) over (4.15.0.134.121) ... 2021-02-11T06:01:06.351 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../4-linux-image-generic_4.15.0.135.122_amd64.deb ... 2021-02-11T06:01:06.351 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking linux-image-generic (4.15.0.135.122) over (4.15.0.134.121) ... 2021-02-11T06:01:06.351 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package linux-headers-4.15.0-135. 2021-02-11T06:01:06.351 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../5-linux-headers-4.15.0-135_4.15.0-135.139_all.deb ... 2021-02-11T06:01:06.351 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking linux-headers-4.15.0-135 (4.15.0-135.139) ... 2021-02-11T06:01:06.351 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package linux-headers-4.15.0-135-generic. 2021-02-11T06:01:06.351 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../6-linux-headers-4.15.0-135-generic_4.15.0-135.139_amd64.deb ... 2021-02-11T06:01:06.352 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking linux-headers-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-11T06:01:06.352 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../7-linux-headers-generic_4.15.0.135.122_amd64.deb ... 2021-02-11T06:01:06.352 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking linux-headers-generic (4.15.0.135.122) over (4.15.0.134.121) ... 2021-02-11T06:01:06.352 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up linux-modules-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-11T06:01:06.352 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up linux-headers-4.15.0-135 (4.15.0-135.139) ... 2021-02-11T06:01:06.352 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up linux-image-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-11T06:01:06.352 INFO:teuthology.orchestra.run.smithi172.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-4.15.0-135-generic 2021-02-11T06:01:06.352 INFO:teuthology.orchestra.run.smithi172.stdout:I: /initrd.img is now a symlink to boot/initrd.img-4.15.0-135-generic 2021-02-11T06:01:06.352 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up linux-modules-extra-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-11T06:01:06.352 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up linux-headers-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-11T06:01:06.352 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up linux-image-generic (4.15.0.135.122) ... 2021-02-11T06:01:06.352 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up linux-headers-generic (4.15.0.135.122) ... 2021-02-11T06:01:06.352 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up linux-generic (4.15.0.135.122) ... 2021-02-11T06:01:06.353 INFO:teuthology.orchestra.run.smithi172.stdout:Processing triggers for linux-image-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-11T06:01:06.353 INFO:teuthology.orchestra.run.smithi172.stdout:/etc/kernel/postinst.d/initramfs-tools: 2021-02-11T06:01:06.353 INFO:teuthology.orchestra.run.smithi172.stdout:update-initramfs: Generating /boot/initrd.img-4.15.0-135-generic 2021-02-11T06:01:18.320 INFO:teuthology.orchestra.run.smithi172.stdout:/etc/kernel/postinst.d/zz-update-grub: 2021-02-11T06:01:18.321 INFO:teuthology.orchestra.run.smithi172.stdout:Sourcing file `/etc/default/grub' 2021-02-11T06:01:18.350 INFO:teuthology.orchestra.run.smithi172.stdout:Generating grub configuration file ... 2021-02-11T06:01:18.698 INFO:teuthology.orchestra.run.smithi172.stdout:Found linux image: /boot/vmlinuz-4.15.0-135-generic 2021-02-11T06:01:18.738 INFO:teuthology.orchestra.run.smithi172.stdout:Found initrd image: /boot/initrd.img-4.15.0-135-generic 2021-02-11T06:01:19.055 INFO:teuthology.orchestra.run.smithi172.stdout:Found linux image: /boot/vmlinuz-4.15.0-134-generic 2021-02-11T06:01:19.086 INFO:teuthology.orchestra.run.smithi172.stdout:Found initrd image: /boot/initrd.img-4.15.0-134-generic 2021-02-11T06:01:19.959 INFO:teuthology.orchestra.run.smithi172.stdout:done 2021-02-11T06:01:20.354 DEBUG:teuthology.orchestra.run.smithi172:> dpkg -s linux-image-generic 2021-02-11T06:01:20.376 INFO:teuthology.orchestra.run.smithi172.stdout:Package: linux-image-generic 2021-02-11T06:01:20.376 INFO:teuthology.orchestra.run.smithi172.stdout:Status: install ok installed 2021-02-11T06:01:20.376 INFO:teuthology.orchestra.run.smithi172.stdout:Priority: optional 2021-02-11T06:01:20.376 INFO:teuthology.orchestra.run.smithi172.stdout:Section: kernel 2021-02-11T06:01:20.376 INFO:teuthology.orchestra.run.smithi172.stdout:Installed-Size: 17 2021-02-11T06:01:20.376 INFO:teuthology.orchestra.run.smithi172.stdout:Maintainer: Ubuntu Kernel Team 2021-02-11T06:01:20.376 INFO:teuthology.orchestra.run.smithi172.stdout:Architecture: amd64 2021-02-11T06:01:20.376 INFO:teuthology.orchestra.run.smithi172.stdout:Source: linux-meta 2021-02-11T06:01:20.377 INFO:teuthology.orchestra.run.smithi172.stdout:Version: 4.15.0.135.122 2021-02-11T06:01:20.377 INFO:teuthology.orchestra.run.smithi172.stdout:Provides: wireguard-modules (= 1.0.20201112-1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.10) 2021-02-11T06:01:20.377 INFO:teuthology.orchestra.run.smithi172.stdout:Depends: linux-image-4.15.0-135-generic, linux-modules-extra-4.15.0-135-generic, linux-firmware, intel-microcode, amd64-microcode 2021-02-11T06:01:20.377 INFO:teuthology.orchestra.run.smithi172.stdout:Recommends: thermald 2021-02-11T06:01:20.377 INFO:teuthology.orchestra.run.smithi172.stdout:Description: Generic Linux kernel image 2021-02-11T06:01:20.377 INFO:teuthology.orchestra.run.smithi172.stdout: This package will always depend on the latest generic kernel image 2021-02-11T06:01:20.377 INFO:teuthology.orchestra.run.smithi172.stdout: available. 2021-02-11T06:01:20.377 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.15.0-134-generic Expected: 4.15.0-135-generic 2021-02-11T06:01:20.377 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2021-02-11T06:01:20.377 DEBUG:teuthology.task.kernel:src is distro, skipping download 2021-02-11T06:01:20.378 DEBUG:teuthology.task.kernel:src is distro, skipping download 2021-02-11T06:01:20.378 INFO:teuthology.task.kernel:Installing distro kernel on c1.mon.a... 2021-02-11T06:01:20.378 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi061.front.sepia.ceph.com, path=None, version=distro) 2021-02-11T06:01:20.378 DEBUG:teuthology.orchestra.run.smithi061:> sudo apt-get clean 2021-02-11T06:01:20.436 DEBUG:teuthology.orchestra.run.smithi061:> sudo apt-get update 2021-02-11T06:01:20.589 INFO:teuthology.orchestra.run.smithi061.stdout:Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease 2021-02-11T06:01:21.672 INFO:teuthology.orchestra.run.smithi061.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2021-02-11T06:01:21.750 INFO:teuthology.orchestra.run.smithi061.stdout:Hit:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2021-02-11T06:01:22.271 INFO:teuthology.orchestra.run.smithi061.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2021-02-11T06:01:23.796 INFO:teuthology.orchestra.run.smithi061.stdout:Reading package lists... 2021-02-11T06:01:23.843 DEBUG:teuthology.orchestra.run.smithi061:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2021-02-11T06:01:23.928 INFO:teuthology.orchestra.run.smithi061.stdout:Reading package lists... 2021-02-11T06:01:24.194 INFO:teuthology.orchestra.run.smithi061.stdout:Building dependency tree... 2021-02-11T06:01:24.195 INFO:teuthology.orchestra.run.smithi061.stdout:Reading state information... 2021-02-11T06:01:24.282 INFO:teuthology.orchestra.run.smithi061.stderr:E: Unable to locate package linux-image-current-generic 2021-02-11T06:01:24.283 DEBUG:teuthology.orchestra.run:got remote process result: 100 2021-02-11T06:01:24.283 DEBUG:teuthology.orchestra.run.smithi061:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2021-02-11T06:01:24.365 INFO:teuthology.orchestra.run.smithi061.stdout:Reading package lists... 2021-02-11T06:01:24.615 INFO:teuthology.orchestra.run.smithi061.stdout:Building dependency tree... 2021-02-11T06:01:24.615 INFO:teuthology.orchestra.run.smithi061.stdout:Reading state information... 2021-02-11T06:01:24.839 INFO:teuthology.orchestra.run.smithi061.stdout:linux-image-generic is already the newest version (4.15.0.135.122). 2021-02-11T06:01:24.840 INFO:teuthology.orchestra.run.smithi061.stdout:0 upgraded, 0 newly installed, 0 to remove and 23 not upgraded. 2021-02-11T06:01:24.841 DEBUG:teuthology.orchestra.run.smithi061:> dpkg -s linux-image-generic 2021-02-11T06:01:24.861 INFO:teuthology.orchestra.run.smithi061.stdout:Package: linux-image-generic 2021-02-11T06:01:24.862 INFO:teuthology.orchestra.run.smithi061.stdout:Status: install ok installed 2021-02-11T06:01:24.862 INFO:teuthology.orchestra.run.smithi061.stdout:Priority: optional 2021-02-11T06:01:24.862 INFO:teuthology.orchestra.run.smithi061.stdout:Section: kernel 2021-02-11T06:01:24.862 INFO:teuthology.orchestra.run.smithi061.stdout:Installed-Size: 17 2021-02-11T06:01:24.862 INFO:teuthology.orchestra.run.smithi061.stdout:Maintainer: Ubuntu Kernel Team 2021-02-11T06:01:24.862 INFO:teuthology.orchestra.run.smithi061.stdout:Architecture: amd64 2021-02-11T06:01:24.862 INFO:teuthology.orchestra.run.smithi061.stdout:Source: linux-meta 2021-02-11T06:01:24.862 INFO:teuthology.orchestra.run.smithi061.stdout:Version: 4.15.0.135.122 2021-02-11T06:01:24.862 INFO:teuthology.orchestra.run.smithi061.stdout:Provides: wireguard-modules (= 1.0.20201112-1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.10) 2021-02-11T06:01:24.862 INFO:teuthology.orchestra.run.smithi061.stdout:Depends: linux-image-4.15.0-135-generic, linux-modules-extra-4.15.0-135-generic, linux-firmware, intel-microcode, amd64-microcode 2021-02-11T06:01:24.862 INFO:teuthology.orchestra.run.smithi061.stdout:Recommends: thermald 2021-02-11T06:01:24.862 INFO:teuthology.orchestra.run.smithi061.stdout:Description: Generic Linux kernel image 2021-02-11T06:01:24.863 INFO:teuthology.orchestra.run.smithi061.stdout: This package will always depend on the latest generic kernel image 2021-02-11T06:01:24.863 INFO:teuthology.orchestra.run.smithi061.stdout: available. 2021-02-11T06:01:24.863 DEBUG:teuthology.orchestra.run.smithi061:> mktemp 2021-02-11T06:01:24.910 INFO:teuthology.orchestra.run.smithi061.stdout:/tmp/tmp.WxfbmnyF1d 2021-02-11T06:01:24.911 DEBUG:teuthology.orchestra.run.smithi061:> sudo cp /boot/grub/grub.cfg /tmp/tmp.WxfbmnyF1d 2021-02-11T06:01:24.965 DEBUG:teuthology.orchestra.run.smithi061:> sudo chmod 0666 /tmp/tmp.WxfbmnyF1d 2021-02-11T06:01:25.080 DEBUG:teuthology.orchestra.remote:smithi061:/tmp/tmp.WxfbmnyF1d is 8KB 2021-02-11T06:01:25.091 DEBUG:teuthology.orchestra.run.smithi061:> rm -fr /tmp/tmp.WxfbmnyF1d 2021-02-11T06:01:25.097 DEBUG:teuthology.orchestra.run.smithi061:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2021-02-11T06:01:25.151 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2021-02-11T06:01:25.151 DEBUG:teuthology.orchestra.run.smithi061:> sudo dd of=/etc/grub.d/01_ceph_kernel 2021-02-11T06:01:25.151 DEBUG:teuthology.orchestra.run.smithi061:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2021-02-11T06:01:25.220 INFO:teuthology.task.kernel:Distro Kernel Version: 4.15.0-135-generic 2021-02-11T06:01:25.221 DEBUG:teuthology.orchestra.run.smithi061:> sudo update-grub 2021-02-11T06:01:26.483 INFO:teuthology.orchestra.run.smithi061.stderr:Sourcing file `/etc/default/grub' 2021-02-11T06:01:26.508 INFO:teuthology.orchestra.run.smithi061.stderr:Generating grub configuration file ... 2021-02-11T06:01:26.786 INFO:teuthology.orchestra.run.smithi061.stderr:Found linux image: /boot/vmlinuz-4.15.0-135-generic 2021-02-11T06:01:26.798 INFO:teuthology.orchestra.run.smithi061.stderr:Found initrd image: /boot/initrd.img-4.15.0-135-generic 2021-02-11T06:01:27.078 INFO:teuthology.orchestra.run.smithi061.stderr:Found linux image: /boot/vmlinuz-4.15.0-134-generic 2021-02-11T06:01:28.177 INFO:teuthology.orchestra.run.smithi061.stderr:Found initrd image: /boot/initrd.img-4.15.0-134-generic 2021-02-11T06:01:28.177 INFO:teuthology.orchestra.run.smithi061.stderr:done 2021-02-11T06:01:28.177 DEBUG:teuthology.orchestra.run.smithi061:> sudo shutdown -r now 2021-02-11T06:01:28.180 INFO:teuthology.task.kernel:Installing distro kernel on c2.mon.a... 2021-02-11T06:01:28.180 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi172.front.sepia.ceph.com, path=None, version=distro) 2021-02-11T06:01:28.180 DEBUG:teuthology.orchestra.run.smithi172:> sudo apt-get clean 2021-02-11T06:01:28.219 DEBUG:teuthology.orchestra.run.smithi172:> sudo apt-get update 2021-02-11T06:01:28.625 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease 2021-02-11T06:01:29.266 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2021-02-11T06:01:29.345 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2021-02-11T06:01:29.425 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2021-02-11T06:01:31.028 INFO:teuthology.orchestra.run.smithi172.stdout:Reading package lists... 2021-02-11T06:01:31.076 DEBUG:teuthology.orchestra.run.smithi172:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2021-02-11T06:01:31.157 INFO:teuthology.orchestra.run.smithi172.stdout:Reading package lists... 2021-02-11T06:01:31.389 INFO:teuthology.orchestra.run.smithi172.stdout:Building dependency tree... 2021-02-11T06:01:31.389 INFO:teuthology.orchestra.run.smithi172.stdout:Reading state information... 2021-02-11T06:01:31.471 INFO:teuthology.orchestra.run.smithi172.stderr:E: Unable to locate package linux-image-current-generic 2021-02-11T06:01:31.472 DEBUG:teuthology.orchestra.run:got remote process result: 100 2021-02-11T06:01:31.472 DEBUG:teuthology.orchestra.run.smithi172:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2021-02-11T06:01:31.552 INFO:teuthology.orchestra.run.smithi172.stdout:Reading package lists... 2021-02-11T06:01:31.796 INFO:teuthology.orchestra.run.smithi172.stdout:Building dependency tree... 2021-02-11T06:01:31.796 INFO:teuthology.orchestra.run.smithi172.stdout:Reading state information... 2021-02-11T06:01:31.993 INFO:teuthology.orchestra.run.smithi172.stdout:linux-image-generic is already the newest version (4.15.0.135.122). 2021-02-11T06:01:31.994 INFO:teuthology.orchestra.run.smithi172.stdout:0 upgraded, 0 newly installed, 0 to remove and 23 not upgraded. 2021-02-11T06:01:31.995 DEBUG:teuthology.orchestra.run.smithi172:> dpkg -s linux-image-generic 2021-02-11T06:01:32.012 INFO:teuthology.orchestra.run.smithi172.stdout:Package: linux-image-generic 2021-02-11T06:01:32.012 INFO:teuthology.orchestra.run.smithi172.stdout:Status: install ok installed 2021-02-11T06:01:32.012 INFO:teuthology.orchestra.run.smithi172.stdout:Priority: optional 2021-02-11T06:01:32.012 INFO:teuthology.orchestra.run.smithi172.stdout:Section: kernel 2021-02-11T06:01:32.013 INFO:teuthology.orchestra.run.smithi172.stdout:Installed-Size: 17 2021-02-11T06:01:32.013 INFO:teuthology.orchestra.run.smithi172.stdout:Maintainer: Ubuntu Kernel Team 2021-02-11T06:01:32.013 INFO:teuthology.orchestra.run.smithi172.stdout:Architecture: amd64 2021-02-11T06:01:32.013 INFO:teuthology.orchestra.run.smithi172.stdout:Source: linux-meta 2021-02-11T06:01:32.013 INFO:teuthology.orchestra.run.smithi172.stdout:Version: 4.15.0.135.122 2021-02-11T06:01:32.013 INFO:teuthology.orchestra.run.smithi172.stdout:Provides: wireguard-modules (= 1.0.20201112-1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.10) 2021-02-11T06:01:32.013 INFO:teuthology.orchestra.run.smithi172.stdout:Depends: linux-image-4.15.0-135-generic, linux-modules-extra-4.15.0-135-generic, linux-firmware, intel-microcode, amd64-microcode 2021-02-11T06:01:32.013 INFO:teuthology.orchestra.run.smithi172.stdout:Recommends: thermald 2021-02-11T06:01:32.013 INFO:teuthology.orchestra.run.smithi172.stdout:Description: Generic Linux kernel image 2021-02-11T06:01:32.013 INFO:teuthology.orchestra.run.smithi172.stdout: This package will always depend on the latest generic kernel image 2021-02-11T06:01:32.013 INFO:teuthology.orchestra.run.smithi172.stdout: available. 2021-02-11T06:01:32.014 DEBUG:teuthology.orchestra.run.smithi172:> mktemp 2021-02-11T06:01:32.059 INFO:teuthology.orchestra.run.smithi172.stdout:/tmp/tmp.6nd7TZNsi5 2021-02-11T06:01:32.059 DEBUG:teuthology.orchestra.run.smithi172:> sudo cp /boot/grub/grub.cfg /tmp/tmp.6nd7TZNsi5 2021-02-11T06:01:32.132 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 0666 /tmp/tmp.6nd7TZNsi5 2021-02-11T06:01:32.249 DEBUG:teuthology.orchestra.remote:smithi172:/tmp/tmp.6nd7TZNsi5 is 8KB 2021-02-11T06:01:32.260 DEBUG:teuthology.orchestra.run.smithi172:> rm -fr /tmp/tmp.6nd7TZNsi5 2021-02-11T06:01:32.266 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2021-02-11T06:01:32.324 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2021-02-11T06:01:32.324 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/grub.d/01_ceph_kernel 2021-02-11T06:01:32.324 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2021-02-11T06:01:32.399 INFO:teuthology.task.kernel:Distro Kernel Version: 4.15.0-135-generic 2021-02-11T06:01:32.399 DEBUG:teuthology.orchestra.run.smithi172:> sudo update-grub 2021-02-11T06:01:33.816 INFO:teuthology.orchestra.run.smithi172.stderr:Sourcing file `/etc/default/grub' 2021-02-11T06:01:33.847 INFO:teuthology.orchestra.run.smithi172.stderr:Generating grub configuration file ... 2021-02-11T06:01:34.065 INFO:teuthology.orchestra.run.smithi172.stderr:Found linux image: /boot/vmlinuz-4.15.0-135-generic 2021-02-11T06:01:34.077 INFO:teuthology.orchestra.run.smithi172.stderr:Found initrd image: /boot/initrd.img-4.15.0-135-generic 2021-02-11T06:01:34.373 INFO:teuthology.orchestra.run.smithi172.stderr:Found linux image: /boot/vmlinuz-4.15.0-134-generic 2021-02-11T06:01:34.387 INFO:teuthology.orchestra.run.smithi172.stderr:Found initrd image: /boot/initrd.img-4.15.0-134-generic 2021-02-11T06:01:34.993 INFO:teuthology.orchestra.run.smithi172.stderr:done 2021-02-11T06:01:34.995 DEBUG:teuthology.orchestra.run.smithi172:> sudo shutdown -r now 2021-02-11T06:02:05.025 INFO:teuthology.misc:Re-opening connections... 2021-02-11T06:02:05.025 INFO:teuthology.misc:trying to connect to ubuntu@smithi061.front.sepia.ceph.com 2021-02-11T06:02:05.025 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-11T06:02:05.026 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-02-11T06:02:23.555 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.61 2021-02-11T06:02:23.555 INFO:teuthology.misc:trying to connect to ubuntu@smithi172.front.sepia.ceph.com 2021-02-11T06:02:23.555 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-11T06:02:23.556 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-02-11T06:02:41.986 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.172 2021-02-11T06:02:41.987 DEBUG:teuthology.misc:waited 36.961703062057495 2021-02-11T06:02:42.987 INFO:teuthology.misc:trying to connect to ubuntu@smithi061.front.sepia.ceph.com 2021-02-11T06:02:42.988 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-11T06:02:42.988 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-02-11T06:03:01.442 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.61 2021-02-11T06:03:01.443 INFO:teuthology.misc:trying to connect to ubuntu@smithi172.front.sepia.ceph.com 2021-02-11T06:03:01.443 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-11T06:03:01.444 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-02-11T06:03:19.874 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.172 2021-02-11T06:03:19.875 DEBUG:teuthology.misc:waited 74.84979462623596 2021-02-11T06:03:20.875 INFO:teuthology.misc:trying to connect to ubuntu@smithi061.front.sepia.ceph.com 2021-02-11T06:03:20.875 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-11T06:03:20.876 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-02-11T06:03:20.955 DEBUG:teuthology.orchestra.run.smithi061:> true 2021-02-11T06:03:21.330 DEBUG:teuthology.misc:waited 76.30459189414978 2021-02-11T06:03:22.331 INFO:teuthology.misc:trying to connect to ubuntu@smithi172.front.sepia.ceph.com 2021-02-11T06:03:22.331 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-11T06:03:22.331 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-02-11T06:03:22.464 DEBUG:teuthology.orchestra.run.smithi172:> true 2021-02-11T06:03:23.201 DEBUG:teuthology.misc:waited 78.17565202713013 2021-02-11T06:03:24.202 INFO:teuthology.task.kernel:Checking client c1.mon.a for new kernel version... 2021-02-11T06:03:24.202 INFO:teuthology.task.kernel:Checking kernel version of c1.mon.a, want "4.15.0-135-generic"... 2021-02-11T06:03:24.203 DEBUG:teuthology.orchestra.run.smithi061:> uname -r 2021-02-11T06:03:24.209 INFO:teuthology.orchestra.run.smithi061.stdout:4.15.0-135-generic 2021-02-11T06:03:24.209 DEBUG:teuthology.task.kernel:current kernel version is 4.15.0-135-generic vs 4.15.0-135-generic 2021-02-11T06:03:24.209 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2021-02-11T06:03:24.209 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2021-02-11T06:03:24.209 INFO:teuthology.task.kernel:Checking client c2.mon.a for new kernel version... 2021-02-11T06:03:24.210 INFO:teuthology.task.kernel:Checking kernel version of c2.mon.a, want "4.15.0-135-generic"... 2021-02-11T06:03:24.210 DEBUG:teuthology.orchestra.run.smithi172:> uname -r 2021-02-11T06:03:24.216 INFO:teuthology.orchestra.run.smithi172.stdout:4.15.0-135-generic 2021-02-11T06:03:24.216 DEBUG:teuthology.task.kernel:current kernel version is 4.15.0-135-generic vs 4.15.0-135-generic 2021-02-11T06:03:24.216 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2021-02-11T06:03:24.217 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2021-02-11T06:03:25.218 INFO:teuthology.task.kernel:Enabling kdb on c1.mon.a... 2021-02-11T06:03:35.885 DEBUG:teuthology.orchestra.run.smithi061:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-02-11T06:03:35.925 INFO:teuthology.orchestra.run.smithi061.stdout:ttyS1 2021-02-11T06:03:35.947 INFO:teuthology.task.kernel:Enabling kdb on c2.mon.a... 2021-02-11T06:03:35.948 DEBUG:teuthology.orchestra.run.smithi172:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-02-11T06:03:35.989 INFO:teuthology.orchestra.run.smithi172.stdout:ttyS1 2021-02-11T06:03:36.010 INFO:teuthology.run_tasks:Running task internal.base... 2021-02-11T06:03:36.018 INFO:teuthology.task.internal:Creating test directory... 2021-02-11T06:03:36.019 DEBUG:teuthology.orchestra.run.smithi061:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-02-11T06:03:36.021 DEBUG:teuthology.orchestra.run.smithi172:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-02-11T06:03:36.058 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2021-02-11T06:03:40.363 INFO:teuthology.run_tasks:Running task internal.archive... 2021-02-11T06:03:40.735 INFO:teuthology.task.internal:Creating archive directory... 2021-02-11T06:03:40.735 DEBUG:teuthology.orchestra.run.smithi061:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-02-11T06:03:40.738 DEBUG:teuthology.orchestra.run.smithi172:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-02-11T06:03:40.818 INFO:teuthology.run_tasks:Running task internal.coredump... 2021-02-11T06:03:40.826 INFO:teuthology.task.internal:Enabling coredump saving... 2021-02-11T06:03:40.826 DEBUG:teuthology.orchestra.run.smithi061:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-02-11T06:03:40.828 DEBUG:teuthology.orchestra.run.smithi172:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-02-11T06:03:40.847 INFO:teuthology.orchestra.run.smithi061.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-02-11T06:03:40.849 INFO:teuthology.orchestra.run.smithi172.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-02-11T06:03:40.854 INFO:teuthology.orchestra.run.smithi061.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-02-11T06:03:40.856 INFO:teuthology.orchestra.run.smithi172.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-02-11T06:03:40.858 INFO:teuthology.run_tasks:Running task internal.sudo... 2021-02-11T06:03:41.508 INFO:teuthology.task.internal:Configuring sudo... 2021-02-11T06:03:41.509 DEBUG:teuthology.orchestra.run.smithi061:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-02-11T06:03:41.511 DEBUG:teuthology.orchestra.run.smithi172:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-02-11T06:03:41.527 INFO:teuthology.run_tasks:Running task internal.syslog... 2021-02-11T06:03:41.655 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2021-02-11T06:03:41.655 DEBUG:teuthology.orchestra.run.smithi061:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-02-11T06:03:41.658 DEBUG:teuthology.orchestra.run.smithi172:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-02-11T06:03:41.664 DEBUG:teuthology.orchestra.run.smithi061:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-02-11T06:03:41.712 DEBUG:teuthology.orchestra.run.smithi061:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-02-11T06:03:41.760 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2021-02-11T06:03:41.760 DEBUG:teuthology.orchestra.run.smithi061:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-02-11T06:03:41.819 DEBUG:teuthology.orchestra.run.smithi172:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-02-11T06:03:41.826 DEBUG:teuthology.orchestra.run.smithi172:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-02-11T06:03:41.873 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2021-02-11T06:03:41.873 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-02-11T06:03:41.951 DEBUG:teuthology.orchestra.run.smithi061:> sudo service rsyslog restart 2021-02-11T06:03:41.953 DEBUG:teuthology.orchestra.run.smithi172:> sudo service rsyslog restart 2021-02-11T06:03:42.040 INFO:teuthology.run_tasks:Running task internal.timer... 2021-02-11T06:03:42.047 INFO:teuthology.task.internal:Starting timer... 2021-02-11T06:03:42.048 INFO:teuthology.run_tasks:Running task pcp... 2021-02-11T06:03:42.372 INFO:teuthology.run_tasks:Running task selinux... 2021-02-11T06:03:43.348 DEBUG:teuthology.task.selinux:Excluding smithi061: OS 'ubuntu' does not support SELinux 2021-02-11T06:03:43.348 DEBUG:teuthology.task.selinux:Excluding smithi172: OS 'ubuntu' does not support SELinux 2021-02-11T06:03:43.348 DEBUG:teuthology.task.selinux:Getting current SELinux state 2021-02-11T06:03:43.348 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2021-02-11T06:03:43.349 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2021-02-11T06:03:43.349 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2021-02-11T06:03:43.523 DEBUG:teuthology.repo_utils:Setting repo remote to git://git.ceph.com/git/ceph-cm-ansible.git 2021-02-11T06:03:43.543 INFO:teuthology.repo_utils:Fetching master from origin 2021-02-11T06:03:43.741 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2021-02-11T06:03:43.954 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}] 2021-02-11T06:03:43.962 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi061.front.sepia.ceph.com,smithi172.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2021-02-11T06:03:44.845 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2021-02-11T06:03:46.769 INFO:teuthology.task.ansible.out:[WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2021-02-11T06:03:56.590 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* Thursday 11 February 2021 06:03:49 +0000 (0:00:00.151) 0:00:00.151 ***** Thursday 11 February 2021 06:03:49 +0000 (0:00:00.017) 0:00:00.169 ***** TASK [Gathering Facts] ********************************************************* ok: [smithi061.front.sepia.ceph.com] ok: [smithi172.front.sepia.ceph.com] Thursday 11 February 2021 06:03:51 +0000 (0:00:01.950) 0:00:02.120 ***** Thursday 11 February 2021 06:03:51 +0000 (0:00:00.031) 0:00:02.151 ***** TASK [ansible-managed : Create the sudo group.] ******************************** ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} Thursday 11 February 2021 06:03:51 +0000 (0:00:00.246) 0:00:02.398 ***** Thursday 11 February 2021 06:03:51 +0000 (0:00:00.031) 0:00:02.430 ***** TASK [ansible-managed : Create the ansible user.] ****************************** changed: [smithi061.front.sepia.ceph.com] => {"append": false, "changed": true, "comment": "cm,,,", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} Thursday 11 February 2021 06:03:52 +0000 (0:00:00.423) 0:00:02.854 ***** changed: [smithi172.front.sepia.ceph.com] => {"append": false, "changed": true, "comment": "cm,,,", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} Thursday 11 February 2021 06:03:52 +0000 (0:00:00.040) 0:00:02.894 ***** TASK [ansible-managed : Delete the ansible users password.] ******************** changed: [smithi172.front.sepia.ceph.com] => {"changed": true, "cmd": ["passwd", "-d", "cm"], "delta": "0:00:00.08 2021-02-11T06:03:56.591 INFO:teuthology.task.ansible.out:5287", "end": "2021-02-11 06:03:52.594092", "rc": 0, "start": "2021-02-11 06:03:52.508805", "stderr": "", "stderr_lines": [], "stdout": "passwd: password expiry information changed.", "stdout_lines": ["passwd: password expiry information changed."]} Thursday 11 February 2021 06:03:52 +0000 (0:00:00.307) 0:00:03.202 ***** changed: [smithi061.front.sepia.ceph.com] => {"changed": true, "cmd": ["passwd", "-d", "cm"], "delta": "0:00:00.140181", "end": "2021-02-11 06:03:52.378305", "rc": 0, "start": "2021-02-11 06:03:52.238124", "stderr": "", "stderr_lines": [], "stdout": "passwd: password expiry information changed.", "stdout_lines": ["passwd: password expiry information changed."]} Thursday 11 February 2021 06:03:52 +0000 (0:00:00.052) 0:00:03.255 ***** TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** ok: [smithi172.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} ok: [smithi061.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} Thursday 11 February 2021 06:03:52 +0000 (0:00:00.229) 0:00:03.484 ***** Thursday 11 February 2021 06:03:52 +0000 (0:00:00.029) 0:00:03.514 ***** TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** changed: [smithi172.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1613023432.9606893-37367428759439/source", "state": "file", "uid": 0} changed: [smithi061.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1 2021-02-11T06:03:56.593 INFO:teuthology.task.ansible.out:613023432.928024-257453396625608/source", "state": "file", "uid": 0} Thursday 11 February 2021 06:03:53 +0000 (0:00:00.620) 0:00:04.134 ***** Thursday 11 February 2021 06:03:53 +0000 (0:00:00.031) 0:00:04.166 ***** TASK [ansible-managed : Add authorized keys for the ansible user.] ************* ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9y 2021-02-11T06:03:56.594 INFO:teuthology.task.ansible.out:RXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h 2021-02-11T06:03:56.595 INFO:teuthology.task.ansible.out:9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb 2021-02-11T06:03:56.597 INFO:teuthology.task.ansible.out:5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} PLAY [teuthology] ************************************************************** skipping: no hosts matched PLAY [testnodes] *************************************************************** Thursday 11 February 2021 06:03:54 +0000 (0:00:00.561) 0:00:04.728 ***** Thursday 11 February 2021 06:03:54 +0000 (0:00:00.036) 0:00:04.765 ***** TASK [Gathering Facts] ********************************************************* ok: [smithi172.front.sepia.ceph.com] [WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. Thursday 11 February 2021 06:03:54 +0000 (0:00:00.738) 0:00:05.503 ***** ok: [smithi061.front.sepia.ceph.com] Thursday 11 February 2021 06:03:55 +0000 (0:00:00.090) 0:00:05.594 ***** TASK [users : Check keys_repo HEAD sha1] *************************************** changed: [smithi061.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.402200", "end": "2021-02-11 06:03:55.599532", "rc": 0, "start": "2021-02-11 06:03:55.197332", "stderr": "", "stderr_lines": [], "stdout": "df8946f08e870e2d2565b79099289ca48bd825f3", "stdout_lines": ["df8946f08e870e2d2565b79099289ca48bd825f3"]} Thursday 11 February 2021 06:03:55 +0000 (0:00:00.635) 0:00:06.229 ***** changed: [smithi172.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote 2021-02-11T06:03:56.598 INFO:teuthology.task.ansible.out:https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.473375", "end": "2021-02-11 06:03:55.609526", "rc": 0, "start": "2021-02-11 06:03:55.136151", "stderr": "", "stderr_lines": [], "stdout": "df8946f08e870e2d2565b79099289ca48bd825f3", "stdout_lines": ["df8946f08e870e2d2565b79099289ca48bd825f3"]} Thursday 11 February 2021 06:03:55 +0000 (0:00:00.043) 0:00:06.272 ***** TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** changed: [smithi061.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.012663", "end": "2021-02-11 06:03:55.526540", "failed_when_result": false, "rc": 0, "start": "2021-02-11 06:03:55.513877", "stderr": "", "stderr_lines": [], "stdout": "0caff10a478f2674721da1965013b225f3ea5800", "stdout_lines": ["0caff10a478f2674721da1965013b225f3ea5800"]} Thursday 11 February 2021 06:03:55 +0000 (0:00:00.135) 0:00:06.408 ***** changed: [smithi172.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.020626", "end": "2021-02-11 06:03:55.838129", "failed_when_result": false, "rc": 0, "start": "2021-02-11 06:03:55.817503", "stderr": "", "stderr_lines": [], "stdout": "0caff10a478f2674721da1965013b225f3ea5800", "stdout_lines": ["0caff10a478f2674721da1965013b225f3ea5800"]} Thursday 11 February 2021 06:03:55 +0000 (0:00:00.041) 0:00:06.449 ***** TASK [users : Determine if we can skip users and pubkeys updates] ************** skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:03:55 +0000 (0:00:00.042) 0:00:06.492 ***** skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:03:55 +0000 (0:00:00.043) 0:00:06.535 ***** TASK [users : Merge extra_admin_users into managed_admin_users] **************** skipping: [s 2021-02-11T06:03:56.599 INFO:teuthology.task.ansible.out:mithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:03:56 +0000 (0:00:00.040) 0:00:06.576 ***** skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:03:56 +0000 (0:00:00.039) 0:00:06.616 ***** TASK [users : Remove managed_admin_users from managed_users] ******************* skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:03:56 +0000 (0:00:00.041) 0:00:06.658 ***** skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:03:56 +0000 (0:00:00.044) 0:00:06.702 ***** TASK [users : Filter the managed_users list] *********************************** skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:03:56 +0000 (0:00:00.041) 0:00:06.743 ***** skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:03:56 +0000 (0:00:00.041) 0:00:06.784 ***** TASK [users : Filter the managed_admin_users list] ***************************** skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:03:56 +0000 (0:00:00.047) 0:00:06.832 ***** skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:03:56 +0000 (0:00:00.044) 0:00:06.877 ***** TASK [users : Set UID_MIN to 1001] ********************************************* ok: [smithi061.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} Thursday 11 February 2021 06:0 2021-02-11T06:03:56.601 INFO:teuthology.task.ansible.out:3:56 +0000 (0:00:00.124) 0:00:07.001 ***** ok: [smithi172.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} Thursday 11 February 2021 06:03:56 +0000 (0:00:00.042) 0:00:07.044 ***** 2021-02-11T06:03:56.605 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2021-02-11T06:03:56.607 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2021-02-11T06:03:56.658 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2021-02-11T06:03:56.743 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2021-02-11T06:03:56.775 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2021-02-11T06:03:56.869 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2021-02-11T06:03:56.911 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2021-02-11T06:03:56.995 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-02-11T06:03:57.058 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-02-11T06:03:57.111 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-02-11T06:03:57.195 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-02-11T06:03:57.238 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-02-11T06:03:57.343 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-02-11T06:03:57.364 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-02-11T06:03:57.479 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-02-11T06:03:57.480 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2021-02-11T06:03:57.616 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-02-11T06:03:57.627 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2021-02-11T06:03:57.742 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-02-11T06:03:57.743 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-02-11T06:03:57.858 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-02-11T06:03:57.869 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-02-11T06:03:57.995 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-02-11T06:03:58.006 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-02-11T06:03:58.122 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-02-11T06:03:58.134 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-02-11T06:04:03.682 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "ap 2021-02-11T06:04:03.683 INFO:teuthology.task.ansible.out:pend": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/ 2021-02-11T06:04:03.685 INFO:teuthology.task.ansible.out:home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": " 2021-02-11T06:04:03.686 INFO:teuthology.task.ansible.out:/bin/bash", "state": "present", "uid": 1116} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": tru 2021-02-11T06:04:03.687 INFO:teuthology.task.ansible.out:e, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f305 2021-02-11T06:04:03.689 INFO:teuthology.task.ansible.out:84f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "move_home": 2021-02-11T06:04:03.690 INFO:teuthology.task.ansible.out: false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} ok: [smithi172.front.sepia.ceph.com] => (ite 2021-02-11T06:04:03.692 INFO:teuthology.task.ansible.out:m={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1123} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcb 2021-02-11T06:04:03.693 INFO:teuthology.task.ansible.out:e6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2021-02-11T06:04:03.693 INFO:teuthology.task.ansible.out: 2021-02-11T06:04:03.695 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-02-11T06:04:03.696 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-02-11T06:04:03.697 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-02-11T06:04:03.698 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-02-11T06:04:03.700 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-02-11T06:04:03.700 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-02-11T06:04:03.701 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-02-11T06:04:03.703 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-02-11T06:04:03.704 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1126} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-02-11T06:04:03.705 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-02-11T06:04:03.707 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-02-11T06:04:03.707 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-02-11T06:04:03.708 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-02-11T06:04:03.709 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-02-11T06:04:03.711 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1132} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-02-11T06:04:03.712 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1133} 2021-02-11T06:04:03.713 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1131} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-02-11T06:04:03.715 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-02-11T06:04:03.716 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-02-11T06:04:03.717 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1133} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-02-11T06:04:03.719 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-02-11T06:04:03.720 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1137} 2021-02-11T06:04:03.720 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-02-11T06:04:03.721 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-02-11T06:04:03.723 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-02-11T06:04:03.724 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1139} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-02-11T06:04:03.725 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1137} 2021-02-11T06:04:03.726 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-02-11T06:04:03.728 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-02-11T06:04:03.729 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1142} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-02-11T06:04:03.730 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-02-11T06:04:03.732 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-02-11T06:04:03.733 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1144} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-02-11T06:04:03.734 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-02-11T06:04:03.736 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-02-11T06:04:03.736 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-02-11T06:04:03.737 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-02-11T06:04:03.738 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-02-11T06:04:03.740 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1144} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-02-11T06:04:03.741 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-02-11T06:04:03.742 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-02-11T06:04:03.744 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1146} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-02-11T06:04:03.745 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-02-11T06:04:03.746 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-02-11T06:04:03.746 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-02-11T06:04:03.748 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-02-11T06:04:03.749 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-02-11T06:04:03.750 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1153} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-02-11T06:04:03.752 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-02-11T06:04:03.753 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-02-11T06:04:03.754 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1151} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-02-11T06:04:03.755 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-02-11T06:04:03.757 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-02-11T06:04:03.757 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-02-11T06:04:03.758 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-02-11T06:04:03.780 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-02-11T06:04:03.874 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-02-11T06:04:03.927 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-02-11T06:04:04.000 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-02-11T06:04:04.053 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-02-11T06:04:04.128 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-02-11T06:04:04.192 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-02-11T06:04:04.254 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-02-11T06:04:04.339 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-02-11T06:04:04.382 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-02-11T06:04:04.477 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-02-11T06:04:04.509 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-02-11T06:04:04.614 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-02-11T06:04:04.636 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-02-11T06:04:04.740 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-02-11T06:04:04.763 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-02-11T06:04:04.764 INFO:teuthology.task.ansible.out: 2021-02-11T06:04:04.879 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-02-11T06:04:04.890 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-02-11T06:04:05.005 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-02-11T06:04:05.017 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-02-11T06:04:05.131 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-02-11T06:04:05.153 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-02-11T06:04:05.258 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-02-11T06:04:05.280 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-02-11T06:04:05.384 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-02-11T06:04:05.406 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-02-11T06:04:05.511 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-02-11T06:04:05.533 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-02-11T06:04:05.638 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-02-11T06:04:05.660 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-02-11T06:04:05.764 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-02-11T06:04:05.775 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-02-11T06:04:05.900 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-02-11T06:04:05.922 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-02-11T06:04:06.027 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-02-11T06:04:06.060 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-02-11T06:04:06.154 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-02-11T06:04:06.207 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-02-11T06:04:06.281 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-02-11T06:04:06.366 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-02-11T06:04:06.685 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1178} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1173} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1179} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group 2021-02-11T06:04:06.687 INFO:teuthology.task.ansible.out:": 1174, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-02-11T06:04:06.692 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-02-11T06:04:06.807 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-02-11T06:04:06.829 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-02-11T06:04:06.944 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-02-11T06:04:06.955 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-02-11T06:04:07.070 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-02-11T06:04:07.081 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1183} 2021-02-11T06:04:07.207 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-02-11T06:04:07.208 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1184} 2021-02-11T06:04:07.333 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-02-11T06:04:07.334 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-02-11T06:04:07.460 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-02-11T06:04:07.471 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-02-11T06:04:07.587 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-02-11T06:04:07.619 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-02-11T06:04:07.713 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-02-11T06:04:07.766 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-02-11T06:04:07.841 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-02-11T06:04:08.479 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1183} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1190} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1184} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1 2021-02-11T06:04:08.481 INFO:teuthology.task.ansible.out:191, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1191} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1185} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1192} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state 2021-02-11T06:04:08.482 INFO:teuthology.task.ansible.out:": "present", "uid": 1193} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1186} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1194} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w 2021-02-11T06:04:08.483 INFO:teuthology.task.ansible.out:19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-02-11T06:04:08.592 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-02-11T06:04:08.593 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-02-11T06:04:08.718 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-02-11T06:04:08.739 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-02-11T06:04:08.843 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-02-11T06:04:08.865 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-02-11T06:04:08.969 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-02-11T06:04:09.012 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-02-11T06:04:09.095 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-02-11T06:04:09.149 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-02-11T06:04:09.234 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-02-11T06:04:21.759 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1193} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1201} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1194} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsri 2021-02-11T06:04:21.761 INFO:teuthology.task.ansible.out:vas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1195} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1202} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae77 2021-02-11T06:04:21.762 INFO:teuthology.task.ansible.out:35b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1196} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1203} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1197} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1204} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "ap 2021-02-11T06:04:21.764 INFO:teuthology.task.ansible.out:pend": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1198} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1205} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1206} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell 2021-02-11T06:04:21.765 INFO:teuthology.task.ansible.out:": "/bin/bash", "state": "present", "uid": 1199} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1207} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1200} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1208} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "h 2021-02-11T06:04:21.766 INFO:teuthology.task.ansible.out:ome": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1201} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1209} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1210} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'zd 2021-02-11T06:04:21.768 INFO:teuthology.task.ansible.out:over', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1202} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1211} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1203} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaL 2021-02-11T06:04:21.769 INFO:teuthology.task.ansible.out:yJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1212} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/b 2021-02-11T06:04:21.769 INFO:teuthology.task.ansible.out:in/bash", "state": "present", "uid": 1204} 2021-02-11T06:04:21.770 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-02-11T06:04:21.772 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-02-11T06:04:21.773 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-02-11T06:04:21.773 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-02-11T06:04:21.774 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-02-11T06:04:21.776 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-02-11T06:04:21.776 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-02-11T06:04:21.777 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-02-11T06:04:21.778 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-02-11T06:04:21.780 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1209} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-02-11T06:04:21.781 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-02-11T06:04:21.783 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1219} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-02-11T06:04:21.784 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-02-11T06:04:21.785 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1212} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-02-11T06:04:21.787 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-02-11T06:04:21.788 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1222} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-02-11T06:04:21.789 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-02-11T06:04:21.791 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-02-11T06:04:21.792 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1215} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-02-11T06:04:21.793 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-02-11T06:04:21.795 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1226} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-02-11T06:04:21.796 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-02-11T06:04:21.797 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-02-11T06:04:21.797 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-02-11T06:04:21.798 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-02-11T06:04:21.800 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1229} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-02-11T06:04:21.801 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1230, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1230} 2021-02-11T06:04:21.802 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-02-11T06:04:21.802 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-02-11T06:04:21.804 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-02-11T06:04:21.805 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/hmunjulu", "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "move_home": false, "name": "hmunjulu", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-02-11T06:04:21.805 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-02-11T06:04:21.806 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-02-11T06:04:21.808 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-02-11T06:04:21.808 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1234, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1234} 2021-02-11T06:04:21.809 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "move_home": false, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-02-11T06:04:21.810 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1225} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1236, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "move_home": false, "name": "adking", "shell": "/bin/bash", "state": "present", "uid": 1236} 2021-02-11T06:04:21.812 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-02-11T06:04:21.813 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1237, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, "move_home": false, "name": "amathuri", "shell": "/bin/bash", "state": "present", "uid": 1237} 2021-02-11T06:04:21.813 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-02-11T06:04:21.814 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-02-11T06:04:21.816 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1229} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1230, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1230} 2021-02-11T06:04:21.817 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2021-02-11T06:04:21.877 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-02-11T06:04:21.878 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/hmunjulu", "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "move_home": false, "name": "hmunjulu", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-02-11T06:04:21.880 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-02-11T06:04:21.881 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1234, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1234} 2021-02-11T06:04:21.881 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "move_home": false, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-02-11T06:04:21.882 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1236, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "move_home": false, "name": "adking", "shell": "/bin/bash", "state": "present", "uid": 1236} 2021-02-11T06:04:21.884 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1237, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, "move_home": false, "name": "amathuri", "shell": "/bin/bash", "state": "present", "uid": 1237} 2021-02-11T06:04:21.884 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2021-02-11T06:04:21.978 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:04:21 +0000 (0:00:25.502) 0:00:32.546 ***** 2021-02-11T06:04:22.011 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:04:22 +0000 (0:00:00.033) 0:00:32.580 ***** 2021-02-11T06:04:22.017 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2021-02-11T06:04:22.054 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:04:22 +0000 (0:00:00.043) 0:00:32.623 ***** 2021-02-11T06:04:22.099 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:04:22 +0000 (0:00:00.044) 0:00:32.667 ***** 2021-02-11T06:04:22.107 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* 2021-02-11T06:04:22.110 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@fedora 2021-02-11T06:04:22.111 INFO:teuthology.task.ansible.out:rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265 2021-02-11T06:04:22.112 INFO:teuthology.task.ansible.out:c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancic 2021-02-11T06:04:22.114 INFO:teuthology.task.ansible.out:h", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1Wi 2021-02-11T06:04:22.115 INFO:teuthology.task.ansible.out:RenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vum 2021-02-11T06:04:22.116 INFO:teuthology.task.ansible.out:rao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69 2021-02-11T06:04:22.118 INFO:teuthology.task.ansible.out:mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d8 2021-02-11T06:04:22.119 INFO:teuthology.task.ansible.out:41a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveirada 2021-02-11T06:04:22.120 INFO:teuthology.task.ansible.out:n@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOe 2021-02-11T06:04:22.121 INFO:teuthology.task.ansible.out:nVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9 2021-02-11T06:04:22.123 INFO:teuthology.task.ansible.out:gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": 2021-02-11T06:04:22.124 INFO:teuthology.task.ansible.out:"dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NI 2021-02-11T06:04:22.125 INFO:teuthology.task.ansible.out:Nj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}]}, "changed": false} 2021-02-11T06:04:22.165 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:04:22 +0000 (0:00:00.065) 0:00:32.733 ***** 2021-02-11T06:04:22.175 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@fedora 2021-02-11T06:04:22.176 INFO:teuthology.task.ansible.out:rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265 2021-02-11T06:04:22.177 INFO:teuthology.task.ansible.out:c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancic 2021-02-11T06:04:22.179 INFO:teuthology.task.ansible.out:h", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1Wi 2021-02-11T06:04:22.180 INFO:teuthology.task.ansible.out:RenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdi 2021-02-11T06:04:22.180 INFO:teuthology.task.ansible.out:llaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vum 2021-02-11T06:04:22.181 INFO:teuthology.task.ansible.out:rao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69 2021-02-11T06:04:22.183 INFO:teuthology.task.ansible.out:mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d8 2021-02-11T06:04:22.184 INFO:teuthology.task.ansible.out:41a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveirada 2021-02-11T06:04:22.185 INFO:teuthology.task.ansible.out:n@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOe 2021-02-11T06:04:22.187 INFO:teuthology.task.ansible.out:nVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9 2021-02-11T06:04:22.188 INFO:teuthology.task.ansible.out:gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": 2021-02-11T06:04:22.189 INFO:teuthology.task.ansible.out:"dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jH 2021-02-11T06:04:22.190 INFO:teuthology.task.ansible.out:iMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NI 2021-02-11T06:04:22.191 INFO:teuthology.task.ansible.out:Nj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}]}, "changed": false} 2021-02-11T06:04:22.231 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:04:22 +0000 (0:00:00.066) 0:00:32.799 ***** 2021-02-11T06:04:23.200 INFO:teuthology.task.ansible.out:FAILED - RETRYING: Clone the keys repo (5 retries left). 2021-02-11T06:04:23.222 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2021-02-11T06:04:23.223 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com -> localhost] => {"after": "df8946f08e870e2d2565b79099289ca48bd825f3", "attempts": 1, "before": "df8946f08e870e2d2565b79099289ca48bd825f3", "changed": false, "remote_url_changed": false} 2021-02-11T06:04:23.260 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:04:23 +0000 (0:00:01.029) 0:00:33.829 ***** 2021-02-11T06:04:23.463 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-11T06:04:23.464 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-02-11T06:04:23.598 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-02-11T06:04:23.723 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-02-11T06:04:23.859 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2021-02-11T06:04:23.860 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2021-02-11T06:04:23.994 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-02-11T06:04:24.139 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2021-02-11T06:04:24.141 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2021-02-11T06:04:24.274 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-02-11T06:04:24.421 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-02-11T06:04:24.547 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-02-11T06:04:24.547 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:04:24.682 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2021-02-11T06:04:24.683 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2021-02-11T06:04:24.830 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2021-02-11T06:04:24.831 INFO:teuthology.task.ansible.out:son", "validate_certs": true} 2021-02-11T06:04:24.841 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:04:24.841 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:04:24.987 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-02-11T06:04:25.112 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2021-02-11T06:04:25.113 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-02-11T06:04:25.227 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-02-11T06:04:25.372 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "davidz", "validate_certs": true} 2021-02-11T06:04:25.508 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-02-11T06:04:25.653 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-02-11T06:04:25.778 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2021-02-11T06:04:25.923 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2021-02-11T06:04:25.924 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2021-02-11T06:04:26.059 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2021-02-11T06:04:26.061 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2021-02-11T06:04:26.216 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-02-11T06:04:26.932 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW 2021-02-11T06:04:26.934 INFO:teuthology.task.ansible.out:4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": fal 2021-02-11T06:04:26.935 INFO:teuthology.task.ansible.out:se, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz 2021-02-11T06:04:26.936 INFO:teuthology.task.ansible.out:1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-02-11T06:04:27.040 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-02-11T06:04:27.041 INFO:teuthology.task.ansible.out: 2021-02-11T06:04:27.176 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-02-11T06:04:27.322 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2021-02-11T06:04:27.324 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2021-02-11T06:04:27.468 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2021-02-11T06:04:27.603 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-02-11T06:04:27.741 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-02-11T06:04:27.866 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-02-11T06:04:28.014 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2021-02-11T06:04:28.154 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-02-11T06:04:28.535 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff 2021-02-11T06:04:28.536 INFO:teuthology.task.ansible.out:992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-02-11T06:04:28.550 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-02-11T06:04:28.675 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2021-02-11T06:04:28.677 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-02-11T06:04:28.821 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2021-02-11T06:04:28.822 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-02-11T06:04:28.937 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-02-11T06:04:29.082 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-02-11T06:04:29.217 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-02-11T06:04:29.331 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": nu 2021-02-11T06:04:29.333 INFO:teuthology.task.ansible.out:ll, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-02-11T06:04:29.466 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2021-02-11T06:04:29.467 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2021-02-11T06:04:29.469 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-02-11T06:04:29.623 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-02-11T06:04:30.478 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "pr 2021-02-11T06:04:30.479 INFO:teuthology.task.ansible.out:esent", "user": "rdias", "validate_certs": true} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2b 2021-02-11T06:04:30.481 INFO:teuthology.task.ansible.out:iUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR 2021-02-11T06:04:30.482 INFO:teuthology.task.ansible.out:26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-02-11T06:04:30.563 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-02-11T06:04:30.729 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2021-02-11T06:04:30.731 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2021-02-11T06:04:30.854 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-02-11T06:04:30.993 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-02-11T06:04:31.160 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2021-02-11T06:04:31.162 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2021-02-11T06:04:31.297 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-02-11T06:04:31.444 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-02-11T06:04:31.590 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-02-11T06:04:31.715 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-02-11T06:04:31.851 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2021-02-11T06:04:31.987 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2021-02-11T06:04:32.102 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2021-02-11T06:04:32.227 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-02-11T06:04:32.372 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2021-02-11T06:04:32.518 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2021-02-11T06:04:32.643 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-02-11T06:04:32.779 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-02-11T06:04:32.926 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-02-11T06:04:33.061 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2021-02-11T06:04:33.196 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2021-02-11T06:04:33.332 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-02-11T06:04:33.457 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-02-11T06:04:33.605 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-02-11T06:04:33.720 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2021-02-11T06:04:33.722 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com -> localhost] => {"after": "df8946f08e870e2d2565b79099289ca48bd825f3", "attempts": 2, "before": "df8946f08e870e2d2565b79099289ca48bd825f3", "changed": false, "remote_url_changed": false} 2021-02-11T06:04:33.768 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:04:33 +0000 (0:00:10.507) 0:00:44.336 ***** 2021-02-11T06:04:33.776 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-11T06:04:33.777 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-02-11T06:04:33.893 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2021-02-11T06:04:33.956 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-02-11T06:04:34.030 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1 2021-02-11T06:04:34.031 INFO:teuthology.task.ansible.out:lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2021-02-11T06:04:34.082 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-02-11T06:04:34.166 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-02-11T06:04:34.208 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-02-11T06:04:34.312 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-02-11T06:04:34.334 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2021-02-11T06:04:34.335 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2021-02-11T06:04:34.449 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-02-11T06:04:34.460 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-02-11T06:04:34.585 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2021-02-11T06:04:34.587 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2021-02-11T06:04:34.596 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2021-02-11T06:04:34.711 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-02-11T06:04:34.732 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.locald 2021-02-11T06:04:34.734 INFO:teuthology.task.ansible.out:omain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2021-02-11T06:04:34.827 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-02-11T06:04:34.879 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2021-02-11T06:04:34.881 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2021-02-11T06:04:34.953 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-02-11T06:04:34.963 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:04:35.005 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-02-11T06:04:35.090 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2021-02-11T06:04:35.091 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2021-02-11T06:04:35.144 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-02-11T06:04:35.217 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2021-02-11T06:04:35.218 INFO:teuthology.task.ansible.out:son", "validate_certs": true} 2021-02-11T06:04:35.219 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:04:35.229 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:04:35.261 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2021-02-11T06:04:35.355 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-02-11T06:04:35.387 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2021-02-11T06:04:35.481 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2021-02-11T06:04:35.483 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-02-11T06:04:35.514 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2021-02-11T06:04:35.597 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-02-11T06:04:35.639 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2021-02-11T06:04:35.724 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "davidz", "validate_certs": true} 2021-02-11T06:04:35.767 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2021-02-11T06:04:35.851 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-02-11T06:04:35.894 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2021-02-11T06:04:45.425 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kD 2021-02-11T06:04:45.426 INFO:teuthology.task.ansible.out:ifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ans 2021-02-11T06:04:45.428 INFO:teuthology.task.ansible.out:ible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHI 2021-02-11T06:04:45.429 INFO:teuthology.task.ansible.out:Ft4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, " 2021-02-11T06:04:45.431 INFO:teuthology.task.ansible.out:item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFt 2021-02-11T06:04:45.432 INFO:teuthology.task.ansible.out:gXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBYB0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2E 2021-02-11T06:04:45.433 INFO:teuthology.task.ansible.out:AAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_op 2021-02-11T06:04:45.433 INFO:teuthology.task.ansible.out:tions": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-02-11T06:04:45.435 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2021-02-11T06:04:45.436 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-02-11T06:04:45.436 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "r 2021-02-11T06:04:45.437 INFO:teuthology.task.ansible.out:yg-", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax 2021-02-11T06:04:45.439 INFO:teuthology.task.ansible.out:/32Hq0dz7sAawlQFCJYPHzVzjwmih46TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB 2021-02-11T06:04:45.440 INFO:teuthology.task.ansible.out:5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAA 2021-02-11T06:04:45.441 INFO:teuthology.task.ansible.out:ABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABA 2021-02-11T06:04:45.443 INFO:teuthology.task.ansible.out:AACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakM 2021-02-11T06:04:45.444 INFO:teuthology.task.ansible.out:EEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94 2021-02-11T06:04:45.445 INFO:teuthology.task.ansible.out:prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoN 2021-02-11T06:04:45.447 INFO:teuthology.task.ansible.out:ddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": " 2021-02-11T06:04:45.448 INFO:teuthology.task.ansible.out:present", "user": "joao", "validate_certs": true} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf374 2021-02-11T06:04:45.449 INFO:teuthology.task.ansible.out:80\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQvUxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRL 2021-02-11T06:04:45.451 INFO:teuthology.task.ansible.out:zxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3 2021-02-11T06:04:45.452 INFO:teuthology.task.ansible.out:N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kach 2021-02-11T06:04:45.453 INFO:teuthology.task.ansible.out:au/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "k 2021-02-11T06:04:45.455 INFO:teuthology.task.ansible.out:ey_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOX 2021-02-11T06:04:45.456 INFO:teuthology.task.ansible.out:EDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79 2021-02-11T06:04:45.457 INFO:teuthology.task.ansible.out:Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/F 2021-02-11T06:04:45.459 INFO:teuthology.task.ansible.out:SwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd 2021-02-11T06:04:45.460 INFO:teuthology.task.ansible.out:5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", " 2021-02-11T06:04:45.462 INFO:teuthology.task.ansible.out:validate_certs": true} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2 2021-02-11T06:04:45.463 INFO:teuthology.task.ansible.out:NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrt 2021-02-11T06:04:45.464 INFO:teuthology.task.ansible.out:Uz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez 2021-02-11T06:04:45.466 INFO:teuthology.task.ansible.out:00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni 2021-02-11T06:04:45.467 INFO:teuthology.task.ansible.out:@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c31 2021-02-11T06:04:45.468 INFO:teuthology.task.ansible.out:0fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EA 2021-02-11T06:04:45.469 INFO:teuthology.task.ansible.out:AAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyKz2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKej 2021-02-11T06:04:45.471 INFO:teuthology.task.ansible.out:Xv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjLd+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb 2021-02-11T06:04:45.472 INFO:teuthology.task.ansible.out:1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo 2021-02-11T06:04:45.473 INFO:teuthology.task.ansible.out:90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false 2021-02-11T06:04:45.475 INFO:teuthology.task.ansible.out:, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state 2021-02-11T06:04:45.476 INFO:teuthology.task.ansible.out:": "present", "user": "mlatimer", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcf 2021-02-11T06:04:45.477 INFO:teuthology.task.ansible.out:D6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc 2021-02-11T06:04:45.479 INFO:teuthology.task.ansible.out:0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjEfdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvY 2021-02-11T06:04:45.480 INFO:teuthology.task.ansible.out:Y8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+dTzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7 2021-02-11T06:04:45.481 INFO:teuthology.task.ansible.out:e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWV 2021-02-11T06:04:45.483 INFO:teuthology.task.ansible.out:k3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr 2021-02-11T06:04:45.484 INFO:teuthology.task.ansible.out:13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-02-11T06:04:45.484 INFO:teuthology.task.ansible.out: 2021-02-11T06:04:45.486 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-02-11T06:04:45.486 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kch 2021-02-11T06:04:45.487 INFO:teuthology.task.ansible.out:ang", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-02-11T06:04:45.488 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iB 2021-02-11T06:04:45.490 INFO:teuthology.task.ansible.out:HuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1 2021-02-11T06:04:45.491 INFO:teuthology.task.ansible.out:aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWp 2021-02-11T06:04:45.492 INFO:teuthology.task.ansible.out:dq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBw 2021-02-11T06:04:45.493 INFO:teuthology.task.ansible.out:sw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQKiUAlB8dxudAY60M7jnIJvs9yb2pNaT+BVv1Vk1kMs0XjPjcw9cyT3DtQ1xGLF367phRi/cfQbXOYbHG+uyBzQ4EEnBFvziktWRx4mfeWYxKjkjbfkVfpKPsxztqCbwm6+NYzMxGK2cExN7jUHnghHAjfA0IUpVVg9peBD3kUveLwS+UxuJAnuj+G4480LS6GDyLrhFEXU7uOIIlW4S9oLnU6vJL6GAzwyLE4ijTA8Ek03SJ01Ey/gpU9AqnQNp9u12l/7sJely6sI9pAuIzmS8Ht324hexxSWN5sFjXgLvLfFK83ziDBdrFBr0WXnWsFdRpHs2CwSq1JTkByqbN hmunjulu@hmunjulu.remote.csb", "key_options": null, "keyfile": "/home/hmunjulu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hmunjulu", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3K 2021-02-11T06:04:45.495 INFO:teuthology.task.ansible.out:yrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAA 2021-02-11T06:04:45.496 INFO:teuthology.task.ansible.out:EAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gabrioux", "validate_certs": true} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "key": "ssh 2021-02-11T06:04:45.497 INFO:teuthology.task.ansible.out:-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev", "key_options": null, "keyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow 2021-02-11T06:04:45.499 INFO:teuthology.task.ansible.out:": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusi 2021-02-11T06:04:45.500 INFO:teuthology.task.ansible.out:ve": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3Nza 2021-02-11T06:04:45.501 INFO:teuthology.task.ansible.out:C1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kkke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present 2021-02-11T06:04:45.502 INFO:teuthology.task.ansible.out:", "user": "jcollin", "validate_certs": true} ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com", "key_options": null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5b 2021-02-11T06:04:45.504 INFO:teuthology.task.ansible.out:ViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} changed: [smithi172.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDr6H7hmtGCUrO3GppgMHwbmA4XT/Vipz/uLG/N8h6ZxtaLQKpCefRpdKcFlOdDwxkTmT5q9vmXQ8nl95lTGq+eeMxl4ho104ZD9e9AIYmgYqFy7/qNP8gkqduIPlypLL9m5rElTcGd7zqfMqgljMvAmnXLtZNJlIuFtszVU5WPI1QCwfQ3t5aHF6j66bpVoN9uWItVGB1LApQz9TtViWsQ9Hl8X4YauJToO+TkuSX+H/OGH7ZC6CPOzNZb3b3vF/jLMnTzXl86PJs1se4DjKkRGLVCVbr2WhLwQeZ8TEn3Hg1Gn5X8zNM5cTo6U8SnXBAe9ZYbuyfvghrNLdlZpmN+8l0/l5zC3z+N+sEzklXwHw4MQLF+KSloVFd9NN/kyenufTZAyp8VxllZ9TgdC+CPORIWyzmu1elOtYo7LNuE6CIh/Zm9B54SO6hVHhVoOCqe/dIyjeHJA0WKIZblK1XZEvWuu0E2T3jhxfe1GICyTukeB8WoBQH9DZ1yzMPt59E= nia@redhat.com", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2021-02-11T06:04:45.545 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-02-11T06:04:45.545 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-02-11T06:04:45.547 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKeh 2021-02-11T06:04:45.548 INFO:teuthology.task.ansible.out:vFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, 2021-02-11T06:04:45.549 INFO:teuthology.task.ansible.out:"comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-02-11T06:04:45.549 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": 2021-02-11T06:04:45.551 INFO:teuthology.task.ansible.out: "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comme 2021-02-11T06:04:45.552 INFO:teuthology.task.ansible.out:nt": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", 2021-02-11T06:04:45.554 INFO:teuthology.task.ansible.out:"changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/au 2021-02-11T06:04:45.555 INFO:teuthology.task.ansible.out:thorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4 2021-02-11T06:04:45.556 INFO:teuthology.task.ansible.out:F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, 2021-02-11T06:04:45.557 INFO:teuthology.task.ansible.out:"path": null, "state": "present", "user": "pnawracay", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlK 2021-02-11T06:04:45.559 INFO:teuthology.task.ansible.out:MSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== 2021-02-11T06:04:45.560 INFO:teuthology.task.ansible.out:sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuval 2021-02-11T06:04:45.562 INFO:teuthology.task.ansible.out:if@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dde 2021-02-11T06:04:45.563 INFO:teuthology.task.ansible.out:d0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a 2021-02-11T06:04:45.564 INFO:teuthology.task.ansible.out:2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7S 2021-02-11T06:04:45.566 INFO:teuthology.task.ansible.out:SPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.localdomain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9b 2021-02-11T06:04:45.567 INFO:teuthology.task.ansible.out:bbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": t 2021-02-11T06:04:45.568 INFO:teuthology.task.ansible.out:rue} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa9 2021-02-11T06:04:45.570 INFO:teuthology.task.ansible.out:7748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": nu 2021-02-11T06:04:45.571 INFO:teuthology.task.ansible.out:ll, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain 2021-02-11T06:04:45.572 INFO:teuthology.task.ansible.out:\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} ok: [smithi061.front.sepia.ceph.c 2021-02-11T06:04:45.574 INFO:teuthology.task.ansible.out:om] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcT 2021-02-11T06:04:45.575 INFO:teuthology.task.ansible.out:oIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.local", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2021-02-11T06:04:45.607 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:04:45 +0000 (0:00:11.839) 0:00:56.176 ***** 2021-02-11T06:04:45.647 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-02-11T06:04:45.648 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:04:45.658 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:04:45.669 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:04:45.669 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:04:45.680 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:04:45.691 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:04:45.691 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-11T06:04:45.692 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2021-02-11T06:04:45.693 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:04:45.703 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:04:45.714 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:04:45.819 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-11T06:04:45.820 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2021-02-11T06:04:45.840 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-02-11T06:04:45.842 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2021-02-11T06:04:45.852 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:04:45.862 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:04:45.946 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-11T06:04:45.947 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2021-02-11T06:04:45.991 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-02-11T06:04:45.995 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2021-02-11T06:04:46.085 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-11T06:04:50.059 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** ok: [smithi172.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0 2021-02-11T06:04:50.060 INFO:teuthology.task.ansible.out:KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "ch 2021-02-11T06:04:50.062 INFO:teuthology.task.ansible.out:anged": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman 2021-02-11T06:04:50.063 INFO:teuthology.task.ansible.out:@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "ite 2021-02-11T06:04:50.064 INFO:teuthology.task.ansible.out:m": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'tamil', 2021-02-11T06:04:50.065 INFO:teuthology.task.ansible.out:'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@ye 2021-02-11T06:04:50.067 INFO:teuthology.task.ansible.out:huda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:04:50.067 INFO:teuthology.task.ansible.out: 2021-02-11T06:04:50.068 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:04:50.070 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:04:50.070 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJi 2021-02-11T06:04:50.071 INFO:teuthology.task.ansible.out:hFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:04:50.073 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:04:50.074 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:04:50.075 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:04:50.076 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:04:50.078 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:04:50.079 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-11T06:04:50.079 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU 2021-02-11T06:04:50.080 INFO:teuthology.task.ansible.out:0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "C 2021-02-11T06:04:50.082 INFO:teuthology.task.ansible.out:onditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:04:50.083 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:04:50.084 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:04:50.085 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:04:50.086 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:04:50.087 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:04:50.095 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} TASK [users : Update au 2021-02-11T06:04:50.096 INFO:teuthology.task.ansible.out:thorized_keys for each user with literal keys] ********** skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": fal 2021-02-11T06:04:50.098 INFO:teuthology.task.ansible.out:se, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovp 2021-02-11T06:04:50.099 INFO:teuthology.task.ansible.out:n': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansibl 2021-02-11T06:04:50.100 INFO:teuthology.task.ansible.out:e_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "cha 2021-02-11T06:04:50.101 INFO:teuthology.task.ansible.out:nged": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8f 2021-02-11T06:04:50.103 INFO:teuthology.task.ansible.out:ddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result 2021-02-11T06:04:50.104 INFO:teuthology.task.ansible.out:was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"nam 2021-02-11T06:04:50.106 INFO:teuthology.task.ansible.out:e": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fc 2021-02-11T06:04:50.107 INFO:teuthology.task.ansible.out:d405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true 2021-02-11T06:04:50.108 INFO:teuthology.task.ansible.out:, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a 2021-02-11T06:04:50.110 INFO:teuthology.task.ansible.out:359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:04:50.111 INFO:teuthology.task.ansible.out: skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": 2021-02-11T06:04:50.112 INFO:teuthology.task.ansible.out:"vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:04:50.113 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:04:50.115 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:04:50.116 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:04:50.117 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:04:50.119 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:04:50.120 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-11T06:04:50.120 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2021-02-11T06:04:50.121 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:04:50.122 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:04:50.123 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:04:50.124 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:04:50.142 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2021-02-11T06:04:50.144 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@loca 2021-02-11T06:04:50.145 INFO:teuthology.task.ansible.out:lhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf8 2021-02-11T06:04:50.146 INFO:teuthology.task.ansible.out:8ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPR 2021-02-11T06:04:50.147 INFO:teuthology.task.ansible.out:otvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut 2021-02-11T06:04:50.149 INFO:teuthology.task.ansible.out:+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "valida 2021-02-11T06:04:50.150 INFO:teuthology.task.ansible.out:te_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnu 2021-02-11T06:04:50.151 INFO:teuthology.task.ansible.out:L8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1 2021-02-11T06:04:50.153 INFO:teuthology.task.ansible.out:+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjLd+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'na 2021-02-11T06:04:50.154 INFO:teuthology.task.ansible.out:me': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'mlatimer', 2021-02-11T06:04:50.155 INFO:teuthology.task.ansible.out:'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb 2021-02-11T06:04:50.157 INFO:teuthology.task.ansible.out:9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null 2021-02-11T06:04:50.158 INFO:teuthology.task.ansible.out:, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item 2021-02-11T06:04:50.160 INFO:teuthology.task.ansible.out:={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+ 2021-02-11T06:04:50.161 INFO:teuthology.task.ansible.out:EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2021-02-11T06:04:50.193 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:04:50 +0000 (0:00:04.584) 0:01:00.761 ***** 2021-02-11T06:04:50.201 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-02-11T06:04:50.224 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:03.559 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:04:50 +0000 (0:00:00.074) 0:01:00.836 ***** TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQKiUAlB8dxudAY60M7jnIJvs9yb2pNaT+BVv1Vk1kMs0XjPjcw9cyT3DtQ1xGLF367phRi/cfQbXOYbHG+uyBzQ4EEnBFvziktWRx4mfeWYxKjkjbfkVfpKPsxztqCbwm6+NYzMxGK2cExN7jUHnghHAjfA0IUpVVg9peBD3kUveLwS+UxuJAnuj+G4480LS6GDyLrhFEXU7uOIIlW4S9oLnU6vJL6GAzwyLE4ijTA8Ek03SJ01Ey/gpU9AqnQNp9u12l/7sJely6sI9pAuIzmS8Ht324hexxSWN5sFjXgLvLfFK83ziDBdrFBr0WXnWsFd 2021-02-11T06:05:03.561 INFO:teuthology.task.ansible.out:RpHs2CwSq1JTkByqbN hmunjulu@hmunjulu.remote.csb", "key_options": null, "keyfile": "/home/hmunjulu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hmunjulu", "validate_certs": true} TASK [users : Remove revoked users] ******************************************** ok: [smithi172.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} TASK [users : Remove revoked users] ******************************************** ok: [smithi172.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} ok: [smithi172.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} TASK [users : Update auth 2021-02-11T06:05:03.562 INFO:teuthology.task.ansible.out:orized_keys using the keys repo] ********************** ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev", "key_options": null, "keyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} TASK [users : Remove revoked users] ******************************************** ok: [smithi172.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "k 2021-02-11T06:05:03.563 INFO:teuthology.task.ansible.out:ey": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} TASK [users : Remove revoked users] ******************************************** ok: [smithi172.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_key 2021-02-11T06:05:03.565 INFO:teuthology.task.ansible.out:s", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} TASK [users : Remove revoked users] ******************************************** ok: [smithi172.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com", "key_options": null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} TASK [users : Remove revoked users] ******************************************** ok: [smithi172.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": 2021-02-11T06:05:03.566 INFO:teuthology.task.ansible.out:"shylesh", "state": "absent"} TASK [users : Update authorized_keys using the keys repo] ********************** changed: [smithi061.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDr6H7hmtGCUrO3GppgMHwbmA4XT/Vipz/uLG/N8h6ZxtaLQKpCefRpdKcFlOdDwxkTmT5q9vmXQ8nl95lTGq+eeMxl4ho104ZD9e9AIYmgYqFy7/qNP8gkqduIPlypLL9m5rElTcGd7zqfMqgljMvAmnXLtZNJlIuFtszVU5WPI1QCwfQ3t5aHF6j66bpVoN9uWItVGB1LApQz9TtViWsQ9Hl8X4YauJToO+TkuSX+H/OGH7ZC6CPOzNZb3b3vF/jLMnTzXl86PJs1se4DjKkRGLVCVbr2WhLwQeZ8TEn3Hg1Gn5X8zNM5cTo6U8SnXBAe9ZYbuyfvghrNLdlZpmN+8l0/l5zC3z+N+sEzklXwHw4MQLF+KSloVFd9NN/kyenufTZAyp8VxllZ9TgdC+CPORIWyzmu1elOtYo7LNuE6CIh/Zm9B54SO6hVHhVoOCqe/dIyjeHJA0WKIZblK1XZEvWuu0E2T3jhxfe1GICyTukeB8WoBQH9DZ1yzMPt59E= nia@redhat.com", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} Thursday 11 February 2021 06:04:51 +0000 (0:00:01.027) 0:01:01.864 ***** TASK [users : Remove revoked users] ******************************************** ok: [smithi172.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewsc 2021-02-11T06:05:03.567 INFO:teuthology.task.ansible.out:hoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\n 2021-02-11T06:05:03.569 INFO:teuthology.task.ansible.out:dmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7 2021-02-11T06:05:03.570 INFO:teuthology.task.ansible.out:PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"nam 2021-02-11T06:05:03.570 INFO:teuthology.task.ansible.out:e": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:03.571 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T06:05:03.573 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} TASK [users : Update authorized_keys for each user with literal keys] ********** ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manag 2021-02-11T06:05:03.574 INFO:teuthology.task.ansible.out:e_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:03.575 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** ok: [smithi172.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} TASK [users : Update authorized_keys for each user with literal keys] ********** ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile" 2021-02-11T06:05:03.577 INFO:teuthology.task.ansible.out:: "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} TASK [users : Remove revoked users] ******************************************** ok: [smithi172.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-02-11T06:05:03.577 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-02-11T06:05:03.578 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} TASK [users : Remove revoked users] ******************************************** ok: [smithi172.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "ch 2021-02-11T06:05:03.579 INFO:teuthology.task.ansible.out:anged": false, "item": "yguang", "name": "yguang", "state": "absent"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:03.579 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b 2021-02-11T06:05:03.581 INFO:teuthology.task.ansible.out:0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": " 2021-02-11T06:05:03.582 INFO:teuthology.task.ansible.out:item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:03.582 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com 2021-02-11T06:05:03.583 INFO:teuthology.task.ansible.out:] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'ye 2021-02-11T06:05:03.585 INFO:teuthology.task.ansible.out:hudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:03.585 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15d 2021-02-11T06:05:03.586 INFO:teuthology.task.ansible.out:fab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** 2021-02-11T06:05:03.587 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-02-11T06:05:03.588 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:03.588 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:03.589 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:03.591 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:03.592 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:03.593 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:03.595 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** 2021-02-11T06:05:03.596 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:03.597 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:03.599 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:03.600 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:03.601 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:03.603 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** 2021-02-11T06:05:03.603 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-02-11T06:05:03.604 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:03.606 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:03.607 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:03.608 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:03.610 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:03.610 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:03.610 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468f 2021-02-11T06:05:03.612 INFO:teuthology.task.ansible.out:e695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** ok: [smithi172.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:03.613 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:03.614 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:03.614 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7 2021-02-11T06:05:03.615 INFO:teuthology.task.ansible.out:ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@loc 2021-02-11T06:05:03.616 INFO:teuthology.task.ansible.out:alhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:03.618 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:03.619 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** ok: [smithi172.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2021-02-11T06:05:03.621 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:03.621 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:03.622 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:03.622 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:03.624 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:03.625 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:03.626 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:03.628 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** ok: [smithi172.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-02-11T06:05:03.629 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:03.629 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:03.631 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:03.632 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:03.633 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:03.634 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:03.635 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:03.635 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:03.636 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** ok: [smithi172.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:03.638 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:03.639 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:03.640 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:03.656 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T06:05:03.657 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} ok: [smithi172.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-02-11T06:05:03.659 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} ok: [smithi172.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-02-11T06:05:03.660 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} ok: [smithi172.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-02-11T06:05:03.661 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-02-11T06:05:03.662 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-02-11T06:05:03.663 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-02-11T06:05:03.663 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-02-11T06:05:03.665 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} ok: [smithi172.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-02-11T06:05:03.666 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-02-11T06:05:03.666 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-02-11T06:05:03.668 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-02-11T06:05:03.668 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-02-11T06:05:03.669 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-02-11T06:05:03.669 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-02-11T06:05:03.671 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "name": "dorinda", "state": "absent"} 2021-02-11T06:05:03.672 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "zyan", "name": "zyan", "remove": false, "state": "absent"} ok: [smithi172.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-02-11T06:05:03.673 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-02-11T06:05:03.673 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-02-11T06:05:03.675 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-02-11T06:05:03.675 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-02-11T06:05:03.676 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-02-11T06:05:03.676 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-02-11T06:05:03.678 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-02-11T06:05:03.679 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} ok: [smithi172.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-02-11T06:05:03.730 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:03 +0000 (0:00:12.434) 0:01:14.298 ***** 2021-02-11T06:05:03.769 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:03 +0000 (0:00:00.038) 0:01:14.337 ***** 2021-02-11T06:05:03.774 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** 2021-02-11T06:05:03.775 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:03.812 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:03 +0000 (0:00:00.043) 0:01:14.381 ***** 2021-02-11T06:05:03.976 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T06:05:03.977 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2021-02-11T06:05:04.102 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-02-11T06:05:04.123 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2021-02-11T06:05:04.125 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => {"changed": true, "checksum": "5ebc9fcf8859a9c716ab10089a7bec7e44d29368", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "ee06ed012f51c2e3764171d4b9180349", "mode": "0644", "owner": "root", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1613023503.7901502-48528153273206/source", "state": "file", "uid": 0} 2021-02-11T06:05:04.162 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:04 +0000 (0:00:00.349) 0:01:14.730 ***** 2021-02-11T06:05:04.189 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2021-02-11T06:05:04.191 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => { "msg": "Host smithi172.front.sepia.ceph.com is running Ubuntu 18.04 (bionic)" } 2021-02-11T06:05:04.226 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:04 +0000 (0:00:00.064) 0:01:14.794 ***** 2021-02-11T06:05:04.232 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T06:05:04.233 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-02-11T06:05:04.265 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2021-02-11T06:05:04.266 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2021-02-11T06:05:04.298 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:04 +0000 (0:00:00.072) 0:01:14.867 ***** 2021-02-11T06:05:04.323 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2021-02-11T06:05:04.325 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:04.357 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:04 +0000 (0:00:00.059) 0:01:14.926 ***** 2021-02-11T06:05:04.363 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T06:05:04.364 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-02-11T06:05:04.375 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2021-02-11T06:05:04.377 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:04.409 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:04 +0000 (0:00:00.051) 0:01:14.978 ***** 2021-02-11T06:05:04.436 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2021-02-11T06:05:04.437 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:04.474 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:04 +0000 (0:00:00.064) 0:01:15.043 ***** 2021-02-11T06:05:04.480 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T06:05:04.481 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-02-11T06:05:04.503 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2021-02-11T06:05:04.504 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:04.537 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:04 +0000 (0:00:00.062) 0:01:15.105 ***** 2021-02-11T06:05:04.563 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2021-02-11T06:05:04.565 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:04.609 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:04 +0000 (0:00:00.072) 0:01:15.178 ***** 2021-02-11T06:05:04.618 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T06:05:04.619 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-02-11T06:05:04.646 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2021-02-11T06:05:04.648 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:04.677 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:04 +0000 (0:00:00.068) 0:01:15.246 ***** 2021-02-11T06:05:04.704 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2021-02-11T06:05:04.706 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:04.743 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:04 +0000 (0:00:00.065) 0:01:15.311 ***** 2021-02-11T06:05:04.750 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T06:05:04.750 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-02-11T06:05:04.772 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:04.806 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:04 +0000 (0:00:00.062) 0:01:15.374 ***** 2021-02-11T06:05:04.843 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T06:05:04.844 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} TASK [common : Include Red Hat encrypted variables.] *************************** skipping: [smithi172.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi172.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-11T06:05:04.878 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:04 +0000 (0:00:00.072) 0:01:15.447 ***** 2021-02-11T06:05:04.905 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:04.940 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:04 +0000 (0:00:00.061) 0:01:15.508 ***** 2021-02-11T06:05:04.956 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T06:05:04.957 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2021-02-11T06:05:04.968 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:05.003 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:05 +0000 (0:00:00.062) 0:01:15.571 ***** 2021-02-11T06:05:05.029 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** skipping: [smithi172.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-11T06:05:05.062 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:05 +0000 (0:00:00.059) 0:01:15.631 ***** 2021-02-11T06:05:05.079 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T06:05:05.079 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-02-11T06:05:05.091 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:05.125 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:05 +0000 (0:00:00.062) 0:01:15.693 ***** 2021-02-11T06:05:05.151 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2021-02-11T06:05:05.153 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-11T06:05:05.185 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:05 +0000 (0:00:00.060) 0:01:15.754 ***** 2021-02-11T06:05:05.202 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T06:05:05.204 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-02-11T06:05:05.214 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:05.248 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:05 +0000 (0:00:00.063) 0:01:15.817 ***** 2021-02-11T06:05:05.275 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:05.309 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:05 +0000 (0:00:00.060) 0:01:15.877 ***** 2021-02-11T06:05:05.326 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T06:05:05.326 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2021-02-11T06:05:05.337 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:05.371 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:05 +0000 (0:00:00.062) 0:01:15.940 ***** 2021-02-11T06:05:05.398 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2021-02-11T06:05:05.400 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:05.432 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:05 +0000 (0:00:00.060) 0:01:16.000 ***** 2021-02-11T06:05:05.449 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T06:05:05.449 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-02-11T06:05:05.460 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* skipping: [smithi172.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-11T06:05:05.497 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:05 +0000 (0:00:00.064) 0:01:16.065 ***** 2021-02-11T06:05:05.523 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-02-11T06:05:05.525 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:05.557 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:05 +0000 (0:00:00.060) 0:01:16.126 ***** 2021-02-11T06:05:05.564 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T06:05:05.565 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2021-02-11T06:05:05.587 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2021-02-11T06:05:05.588 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:05.621 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:05 +0000 (0:00:00.063) 0:01:16.189 ***** 2021-02-11T06:05:05.647 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:05.681 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:05 +0000 (0:00:00.060) 0:01:16.250 ***** 2021-02-11T06:05:05.688 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T06:05:05.688 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-02-11T06:05:05.710 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:05.744 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:05 +0000 (0:00:00.062) 0:01:16.313 ***** 2021-02-11T06:05:05.771 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:05.804 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:05 +0000 (0:00:00.059) 0:01:16.373 ***** 2021-02-11T06:05:05.810 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T06:05:05.811 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-02-11T06:05:05.833 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:05.867 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:05 +0000 (0:00:00.062) 0:01:16.435 ***** 2021-02-11T06:05:05.893 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2021-02-11T06:05:05.894 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-11T06:05:05.926 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:05 +0000 (0:00:00.059) 0:01:16.494 ***** 2021-02-11T06:05:05.933 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T06:05:05.933 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2021-02-11T06:05:05.955 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:05.989 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:05 +0000 (0:00:00.063) 0:01:16.558 ***** 2021-02-11T06:05:06.027 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2021-02-11T06:05:06.028 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:06.063 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:06 +0000 (0:00:00.073) 0:01:16.632 ***** 2021-02-11T06:05:06.070 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T06:05:06.071 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-02-11T06:05:06.093 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:06.129 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:06 +0000 (0:00:00.065) 0:01:16.697 ***** 2021-02-11T06:05:06.166 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2021-02-11T06:05:06.168 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:06.178 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:06.180 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T06:05:06.181 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-02-11T06:05:06.215 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:06 +0000 (0:00:00.086) 0:01:16.783 ***** 2021-02-11T06:05:06.252 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-02-11T06:05:06.254 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-mirrorlist", "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-testing-mirrorlist", "skip_reason": "Conditional result was False"} 2021-02-11T06:05:06.290 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:06 +0000 (0:00:00.074) 0:01:16.858 ***** 2021-02-11T06:05:06.306 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T06:05:06.306 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-02-11T06:05:06.318 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:06.352 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:06 +0000 (0:00:00.062) 0:01:16.920 ***** 2021-02-11T06:05:06.422 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T06:05:06.423 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-02-11T06:05:06.495 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-02-11T06:05:06.497 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "cmd": "cat /etc/timezone", "delta": "0:00:00.011174", "end": "2021-02-11 06:05:06.466655", "rc": 0, "start": "2021-02-11 06:05:06.455481", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-02-11T06:05:06.528 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:06 +0000 (0:00:00.176) 0:01:17.097 ***** 2021-02-11T06:05:06.544 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T06:05:06.546 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2021-02-11T06:05:06.556 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2021-02-11T06:05:06.557 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:06.589 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:06 +0000 (0:00:00.060) 0:01:17.158 ***** 2021-02-11T06:05:06.615 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:06.649 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:06 +0000 (0:00:00.060) 0:01:17.218 ***** 2021-02-11T06:05:06.667 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** ok: [smithi061.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-02-11T06:05:06.689 INFO:teuthology.task.ansible.out: TASK [common : Mask sleep units] *********************************************** 2021-02-11T06:05:06.690 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "skip_reason": "Conditional result was False"} 2021-02-11T06:05:06.700 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "skip_reason": "Conditional result was False"} 2021-02-11T06:05:06.711 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "skip_reason": "Conditional result was False"} 2021-02-11T06:05:06.747 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:06 +0000 (0:00:00.097) 0:01:17.315 ***** 2021-02-11T06:05:06.773 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-02-11T06:05:06.774 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:06.808 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:06 +0000 (0:00:00.060) 0:01:17.376 ***** 2021-02-11T06:05:06.814 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T06:05:06.815 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-02-11T06:05:06.837 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2021-02-11T06:05:06.838 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:06.870 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:06 +0000 (0:00:00.062) 0:01:17.439 ***** 2021-02-11T06:05:06.897 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2021-02-11T06:05:06.899 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:06.932 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:06 +0000 (0:00:00.061) 0:01:17.500 ***** 2021-02-11T06:05:06.938 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T06:05:06.939 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-02-11T06:05:06.961 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2021-02-11T06:05:06.962 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:06.995 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:06 +0000 (0:00:00.063) 0:01:17.563 ***** 2021-02-11T06:05:07.020 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2021-02-11T06:05:07.022 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:07.055 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:07 +0000 (0:00:00.059) 0:01:17.623 ***** 2021-02-11T06:05:07.060 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T06:05:07.061 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-02-11T06:05:07.155 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-02-11T06:05:07.280 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-02-11T06:05:07.396 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-02-11T06:05:07.524 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-02-11T06:05:08.874 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** changed: [smithi172.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.348731", "end": "2021-02-11 06:05:07.500407", "rc": 0, "start": "2021-02-11 06:05:07.151676", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} Thursday 11 February 2021 06:05:07 +0000 (0:00:00.557) 0:01:18.181 ***** TASK [common : Install Kerberos Packages (RedHat)] ***************************** skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:05:07 +0000 (0:00:00.062) 0:01:18.243 ***** TASK [users : Remove revoked users] ******************************************** ok: [smithi061.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} ok: [smithi061.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} TASK [common : Clean apt cache] ************************************************ [WARNING]: Consider using the apt module rather than running 'apt-get'. If you need to use command because apt is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi172.front.sepia.ceph.com] => {"changed": true, "cmd": ["apt-get", "clean"], "delta": "0:00:00.064632", "end": "2021-02-11 06:05:07.839901", "rc": 0, "start": "2021-02-11 06:05:07.775269", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} Thursday 11 February 2021 06:05:07 +0000 (0:00:00.230) 0:01:18.474 ***** TASK [users : Remove revoked users] ******************************************** ok: [smithi061.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", 2021-02-11T06:05:08.875 INFO:teuthology.task.ansible.out:"changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} ok: [smithi061.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} ok: [smithi061.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} ok: [smithi061.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} ok: [smithi061.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} ok: [smithi061.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "name": "dorinda", "state": "absent"} 2021-02-11T06:05:09.007 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "zyan", "name": "zyan", "remove": false, "state": "absent"} 2021-02-11T06:05:09.132 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-02-11T06:05:09.257 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-02-11T06:05:09.381 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-02-11T06:05:09.496 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-02-11T06:05:09.621 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-02-11T06:05:09.736 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-02-11T06:05:09.861 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-02-11T06:05:09.986 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-02-11T06:05:10.122 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-02-11T06:05:10.123 INFO:teuthology.task.ansible.out: 2021-02-11T06:05:10.238 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-02-11T06:05:10.288 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:10 +0000 (0:00:02.382) 0:01:20.856 ***** 2021-02-11T06:05:10.607 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2021-02-11T06:05:10.609 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => {"changed": true, "checksum": "5ebc9fcf8859a9c716ab10089a7bec7e44d29368", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "ee06ed012f51c2e3764171d4b9180349", "mode": "0644", "owner": "root", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1613023510.3118951-179001851492312/source", "state": "file", "uid": 0} 2021-02-11T06:05:10.639 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:10 +0000 (0:00:00.351) 0:01:21.208 ***** 2021-02-11T06:05:10.774 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2021-02-11T06:05:10.776 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => { "msg": "Host smithi061.front.sepia.ceph.com is running Ubuntu 18.04 (bionic)" } 2021-02-11T06:05:10.806 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:10 +0000 (0:00:00.166) 0:01:21.374 ***** 2021-02-11T06:05:10.842 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2021-02-11T06:05:10.843 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2021-02-11T06:05:10.874 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:10 +0000 (0:00:00.068) 0:01:21.443 ***** 2021-02-11T06:05:10.900 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2021-02-11T06:05:10.901 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:10.932 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:10 +0000 (0:00:00.057) 0:01:21.500 ***** 2021-02-11T06:05:10.957 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2021-02-11T06:05:10.958 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:10.989 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:10 +0000 (0:00:00.057) 0:01:21.557 ***** 2021-02-11T06:05:11.015 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2021-02-11T06:05:11.016 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:11.047 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:11 +0000 (0:00:00.057) 0:01:21.615 ***** 2021-02-11T06:05:11.073 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:11.106 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:11 +0000 (0:00:00.058) 0:01:21.674 ***** 2021-02-11T06:05:11.133 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2021-02-11T06:05:11.134 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:11.165 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:11 +0000 (0:00:00.059) 0:01:21.733 ***** 2021-02-11T06:05:11.190 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2021-02-11T06:05:11.191 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:11.224 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:11 +0000 (0:00:00.058) 0:01:21.792 ***** 2021-02-11T06:05:11.251 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2021-02-11T06:05:11.252 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:11.286 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:11 +0000 (0:00:00.061) 0:01:21.854 ***** 2021-02-11T06:05:11.313 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2021-02-11T06:05:11.314 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:11.345 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:11 +0000 (0:00:00.059) 0:01:21.914 ***** 2021-02-11T06:05:11.382 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2021-02-11T06:05:11.383 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi061.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-11T06:05:11.414 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:11 +0000 (0:00:00.068) 0:01:21.983 ***** 2021-02-11T06:05:11.440 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:11.476 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:11 +0000 (0:00:00.061) 0:01:22.044 ***** 2021-02-11T06:05:11.501 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2021-02-11T06:05:11.503 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:11.534 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:11 +0000 (0:00:00.058) 0:01:22.103 ***** 2021-02-11T06:05:11.560 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2021-02-11T06:05:11.561 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-11T06:05:11.592 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:11 +0000 (0:00:00.057) 0:01:22.160 ***** 2021-02-11T06:05:11.619 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:11.651 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:11 +0000 (0:00:00.058) 0:01:22.219 ***** 2021-02-11T06:05:11.676 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2021-02-11T06:05:11.677 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-11T06:05:11.710 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:11 +0000 (0:00:00.058) 0:01:22.278 ***** 2021-02-11T06:05:11.737 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:11.769 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:11 +0000 (0:00:00.059) 0:01:22.338 ***** 2021-02-11T06:05:11.796 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2021-02-11T06:05:11.797 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:11.829 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:11 +0000 (0:00:00.060) 0:01:22.398 ***** 2021-02-11T06:05:11.855 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2021-02-11T06:05:11.857 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:11.890 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:11 +0000 (0:00:00.060) 0:01:22.458 ***** 2021-02-11T06:05:11.917 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2021-02-11T06:05:11.918 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:11.952 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:11 +0000 (0:00:00.061) 0:01:22.520 ***** 2021-02-11T06:05:11.979 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2021-02-11T06:05:11.980 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-11T06:05:12.014 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:12 +0000 (0:00:00.062) 0:01:22.583 ***** 2021-02-11T06:05:12.041 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-02-11T06:05:12.042 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:12.076 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:12 +0000 (0:00:00.062) 0:01:22.645 ***** 2021-02-11T06:05:12.093 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2021-02-11T06:05:12.095 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"attempts": 1, "cache_update_time": 1613023222, "cache_updated": false, "changed": false} 2021-02-11T06:05:12.130 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:12 +0000 (0:00:00.053) 0:01:22.698 ***** 2021-02-11T06:05:12.136 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2021-02-11T06:05:12.138 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:12.174 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:12 +0000 (0:00:00.044) 0:01:22.742 ***** 2021-02-11T06:05:12.220 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2021-02-11T06:05:12.221 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:12.254 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:12 +0000 (0:00:00.079) 0:01:22.822 ***** 2021-02-11T06:05:12.291 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2021-02-11T06:05:12.293 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:12.327 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:12 +0000 (0:00:00.072) 0:01:22.895 ***** 2021-02-11T06:05:12.354 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2021-02-11T06:05:12.356 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:12.389 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:12 +0000 (0:00:00.062) 0:01:22.957 ***** 2021-02-11T06:05:12.428 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2021-02-11T06:05:12.429 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:12.464 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:12 +0000 (0:00:00.075) 0:01:23.033 ***** 2021-02-11T06:05:12.492 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2021-02-11T06:05:12.494 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-11T06:05:12.526 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:12 +0000 (0:00:00.061) 0:01:23.094 ***** 2021-02-11T06:05:12.553 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2021-02-11T06:05:12.555 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:12.588 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:12 +0000 (0:00:00.062) 0:01:23.157 ***** 2021-02-11T06:05:12.615 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2021-02-11T06:05:12.617 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:12.650 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:12 +0000 (0:00:00.062) 0:01:23.219 ***** 2021-02-11T06:05:12.677 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2021-02-11T06:05:12.679 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:12.711 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:12 +0000 (0:00:00.060) 0:01:23.279 ***** 2021-02-11T06:05:12.753 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2021-02-11T06:05:12.754 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:12.788 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:12 +0000 (0:00:00.077) 0:01:23.357 ***** 2021-02-11T06:05:12.817 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-02-11T06:05:12.818 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-mirrorlist", "skip_reason": "Conditional result was False"} 2021-02-11T06:05:12.819 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-testing-mirrorlist", "skip_reason": "Conditional result was False"} 2021-02-11T06:05:12.852 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:12 +0000 (0:00:00.063) 0:01:23.421 ***** 2021-02-11T06:05:12.880 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:12.914 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:12 +0000 (0:00:00.061) 0:01:23.482 ***** 2021-02-11T06:05:12.986 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2021-02-11T06:05:25.958 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"cache_update_time": 1613023222, "cache_updated": false, "changed": false} Thursday 11 February 2021 06:05:13 +0000 (0:00:00.113) 0:01:23.596 ***** TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:05:13 +0000 (0:00:00.060) 0:01:23.656 ***** TASK [common : Get the current timezone] *************************************** ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "cmd": "cat /etc/timezone", "delta": "0:00:00.013829", "end": "2021-02-11 06:05:12.869127", "rc": 0, "start": "2021-02-11 06:05:12.855298", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} Thursday 11 February 2021 06:05:13 +0000 (0:00:00.056) 0:01:23.713 ***** TASK [common : Set the timezone in /etc/timezone] ****************************** skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:05:13 +0000 (0:00:00.062) 0:01:23.776 ***** TASK [common : Inform the OS of the new timezone] ****************************** skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:05:13 +0000 (0:00:00.062) 0:01:23.838 ***** TASK [common : Mask sleep units] *********************************************** skipping: [smithi061.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (i 2021-02-11T06:05:25.959 INFO:teuthology.task.ansible.out:tem=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:05:13 +0000 (0:00:00.095) 0:01:23.934 ***** TASK [common : Get the current timezone] *************************************** skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:05:13 +0000 (0:00:00.061) 0:01:23.996 ***** TASK [common : Set the timezone] *********************************************** skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:05:13 +0000 (0:00:00.063) 0:01:24.060 ***** TASK [common : Add base OpenSUSE Leap repo] ************************************ skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:05:13 +0000 (0:00:00.062) 0:01:24.123 ***** TASK [common : Add krb5 config file] ******************************************* ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "size": 131, "state": "file", "uid": 0} Thursday 11 February 2021 06:05:13 +0000 (0:00:00.041) 0:01:24.164 ***** TASK [common : Add updates OpenSUSE Leap repo] ********************************* skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:05:13 +0000 (0:00:00.041) 0:01:24.206 ***** TASK [common : 2021-02-11T06:05:25.960 INFO:teuthology.task.ansible.out: Refresh repos] ************************************************** skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:05:13 +0000 (0:00:00.058) 0:01:24.264 ***** TASK [common : Make sure smartmontools is installed] *************************** ok: [smithi172.front.sepia.ceph.com] => {"cache_update_time": 1613023222, "cache_updated": false, "changed": false} Thursday 11 February 2021 06:05:14 +0000 (0:00:00.834) 0:01:25.099 ***** TASK [common : Set the hardware clock] ***************************************** changed: [smithi061.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.917469", "end": "2021-02-11 06:05:14.500441", "rc": 0, "start": "2021-02-11 06:05:13.582972", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} Thursday 11 February 2021 06:05:14 +0000 (0:00:00.249) 0:01:25.348 ***** TASK [common : Install Kerberos Packages (RedHat)] ***************************** skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:05:14 +0000 (0:00:00.059) 0:01:25.408 ***** TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** ok: [smithi172.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "size": 2716736, "state": "file", "uid": 0} TASK [common : Clean apt cache] ************************************************ changed: [smithi061.front.sepia.ceph.com] => {"changed": true, "cmd": ["apt-get", "clean"], "delta": "0:00:00.036939", "end": "2021-02-11 06:05:14.763686", "rc": 0, "start": "2021-02-11 06:05:14.726747", "stde 2021-02-11T06:05:25.962 INFO:teuthology.task.ansible.out:rr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} Thursday 11 February 2021 06:05:15 +0000 (0:00:00.198) 0:01:25.607 ***** TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** ok: [smithi172.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "size": 1952038, "state": "file", "uid": 0} ok: [smithi172.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "size": 580432, "state": "file", "uid": 0} Thursday 11 February 2021 06:05:15 +0000 (0:00:00.417) 0:01:26.024 ***** TASK [common : Create /usr/libexec.] ******************************************* ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "size": 4096, "state": "directory", "uid": 0} Thursday 11 February 2021 06:05:15 +0000 (0:00:00.151) 0:01:26.176 ***** TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** ok: [smithi172.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "size": 13355, "state": "file", "uid": 0} ok: [smithi172.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0 2021-02-11T06:05:25.963 INFO:teuthology.task.ansible.out:, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "size": 8602, "state": "file", "uid": 0} ok: [smithi172.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "size": 3605, "state": "file", "uid": 0} Thursday 11 February 2021 06:05:16 +0000 (0:00:00.799) 0:01:26.976 ***** TASK [common : Include apt_system vars] **************************************** ok: [smithi172.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib/nagios/plugins", "nrpe_group": "nagios", "nrpe_packages": ["nagios-nrpe-server", "nagios-plugins-basic"], "nrpe_service_name": "nagios-nrpe-server", "nrpe_user": "nagios"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/apt_systems.yml"], "changed": false} Thursday 11 February 2021 06:05:16 +0000 (0:00:00.069) 0:01:27.045 ***** TASK [common : Include yum_systems vars] *************************************** skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:05:16 +0000 (0:00:00.059) 0:01:27.105 ***** TASK [common : Check for epel] ************************************************* skipping: [smithi172.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} Thursday 11 February 2021 06:05:16 +0000 (0:00:00.057) 0:01:27.162 ***** TASK [common : Install nrpe without epel] ************************************** skipping: [smithi172.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nag 2021-02-11T06:05:25.964 INFO:teuthology.task.ansible.out:ios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:05:16 +0000 (0:00:00.079) 0:01:27.242 ***** TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:05:16 +0000 (0:00:00.056) 0:01:27.298 ***** TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** ok: [smithi172.front.sepia.ceph.com] => {"cache_update_time": 1613023222, "cache_updated": false, "changed": false} Thursday 11 February 2021 06:05:17 +0000 (0:00:01.054) 0:01:28.353 ***** TASK [common : Upload nagi 2021-02-11T06:05:25.965 INFO:teuthology.task.ansible.out:os sudoers.d for raid utilities.] ******************** ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "checksum": "4880e187ff9f437cc33858eb1d0fa49a06d67be4", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "size": 180, "state": "file", "uid": 0} Thursday 11 February 2021 06:05:18 +0000 (0:00:00.332) 0:01:28.685 ***** TASK [common : Upload nagios check_mem script] ********************************* ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/nagios/plugins/check_mem.sh", "size": 1964, "state": "file", "uid": 0} Thursday 11 February 2021 06:05:18 +0000 (0:00:00.342) 0:01:29.028 ***** TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ ok: [smithi172.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "DAEMON_OPTS", "msg": ""} ok: [smithi172.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "NRPE_OPTS", "msg": ""} Thursday 11 February 2021 06:05:18 +0000 (0:00:00.277) 0:01:29.305 ***** TASK [common : Update apt cache] *********************************************** ok: [smithi061.front.sepia.ceph.com] => {"attempts": 1, "cache_update_time": 1613023167, "cache_updated": false, "changed": false} Thursday 11 February 2021 06:05:18 +0000 (0:00:00.045) 0:01:29.351 ***** TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:05:18 +0000 (0:00:00.041) 0:01:29.392 ***** TASK [common : Check firewalld status] 2021-02-11T06:05:25.967 INFO:teuthology.task.ansible.out: ***************************************** skipping: [smithi172.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} Thursday 11 February 2021 06:05:18 +0000 (0:00:00.059) 0:01:29.452 ***** TASK [common : Open nrpe port if firewalld enabled] **************************** skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:05:18 +0000 (0:00:00.070) 0:01:29.522 ***** TASK [common : Upload nagios nrpe config.] ************************************* ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "checksum": "a0ee4c31e844cf26108115f6af99f92f904eafbf", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "size": 1176, "state": "file", "uid": 0} Thursday 11 February 2021 06:05:19 +0000 (0:00:00.346) 0:01:29.869 ***** TASK [common : Install Kerberos Packages (Debian)] ***************************** ok: [smithi061.front.sepia.ceph.com] => {"cache_update_time": 1613023167, "cache_updated": false, "changed": false} Thursday 11 February 2021 06:05:19 +0000 (0:00:00.394) 0:01:30.264 ***** TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:05:19 +0000 (0:00:00.061) 0:01:30.325 ***** TASK [common : Make sure nagios nrpe service is running.] ********************** ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nagios-nrpe-server", "state": "started", "status": {"ActiveEnterTimestamp": "Thu 2021-02-11 06:03:21 UTC", "ActiveEnterTimestampMonotonic": "18754651", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "remote-fs.target lo 2021-02-11T06:05:25.968 INFO:teuthology.task.ansible.out:cal-fs.target systemd-tmpfiles-setup.service -.mount network.target system.slice nss-lookup.target basic.target time-sync.target var-run.mount sysinit.target systemd-journald.socket", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2021-02-11 06:03:21 UTC", "AssertTimestampMonotonic": "18753636", "Before": "shutdown.target plymouth-quit.service xdm.service getty@tty1.service multi-user.tar 2021-02-11T06:05:25.968 INFO:teuthology.task.ansible.out:get", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2021-02-11 06:03:21 UTC", "ConditionTimestampMonotonic": "18753636", "ConfigurationDirectoryMode": "0755", "Conflicts": "nrpe.socket shutdown.target", "ControlGroup": "/system.slice/nagios-nrpe-server.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EnvironmentFile": "/etc/default/nagios-nrpe-server (ignore_errors=yes)", "ExecMainCode": "0", "Ex 2021-02-11T06:05:25.970 INFO:teuthology.task.ansible.out:ecMainExitTimestampMonotonic": "0", "ExecMainPID": "1298", "ExecMainStartTimestamp": "Thu 2021-02-11 06:03:21 UTC", "ExecMainStartTimestampMonotonic": "18754620", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; ignore_errors=no ; start_time=[Thu 2021-02-11 06:03:21 UTC] ; stop_time=[n/a] ; pid=1298 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nagios/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/nagios-nrpe-server.service", "GID": "118", "Group": "nagios", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nagios-nrpe-server.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2021-02-11 06:03:21 UTC", "InactiveExitTimestampMonotonic": "18754651", "InvocationID": "25cf870fbfa44485a9dfb024b114e0b6", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864" 2021-02-11T06:05:25.971 INFO:teuthology.task.ansible.out:, "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128029", "LimitNPROCSoft": "128029", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "128029", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1298", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nagios-nrpe-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nagios/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice -.mount sysinit.target", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "s 2021-02-11T06:05:25.972 INFO:teuthology.task.ansible.out:ystem.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2021-02-11 06:03:21 UTC", "StateChangeTimestampMonotonic": "18754651", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "109", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "User": "nagios", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2021-02-11 06:03:21 UTC", "WatchdogTimestampMonotonic": "18754650", "WatchdogUSec": "0"}} TASK [common : Add krb5 config file] ******************************************* ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "size": 131, "state": "file", "uid": 0} 2021-02-11T06:05:26.004 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:26 +0000 (0:00:06.247) 0:01:36.573 ***** 2021-02-11T06:05:26.043 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:26 +0000 (0:00:00.038) 0:01:36.612 ***** 2021-02-11T06:05:26.071 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2021-02-11T06:05:26.072 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:26.103 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:26 +0000 (0:00:00.059) 0:01:36.672 ***** 2021-02-11T06:05:26.129 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2021-02-11T06:05:26.131 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:26.161 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:26 +0000 (0:00:00.057) 0:01:36.729 ***** 2021-02-11T06:05:26.187 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2021-02-11T06:05:26.188 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:26.219 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:26 +0000 (0:00:00.057) 0:01:36.787 ***** 2021-02-11T06:05:26.246 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2021-02-11T06:05:26.247 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:26.284 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:26 +0000 (0:00:00.061) 0:01:36.848 ***** 2021-02-11T06:05:26.308 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:26.342 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:26 +0000 (0:00:00.062) 0:01:36.911 ***** 2021-02-11T06:05:27.702 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:05:26 +0000 (0:00:00.062) 0:01:36.973 ***** TASK [common : nrpe - Compile SELinux module file] ***************************** skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:05:26 +0000 (0:00:00.061) 0:01:37.034 ***** TASK [common : nrpe - Build SELinux policy package] **************************** skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:05:26 +0000 (0:00:00.061) 0:01:37.096 ***** TASK [common : nrpe - Load SELinux policy package] ***************************** skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:05:26 +0000 (0:00:00.060) 0:01:37.156 ***** TASK [common : nrpe - Remove temporary files] ********************************** skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:05:26 +0000 (0:00:00.059) 0:01:37.215 ***** TASK [testnode : Include package type specific vars.] ************************** ok: [smithi172.front.sepia.ceph.com] => {"ansible_facts": {"apt_repos": [], "ceph_packages_to_remove": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common"], "common_apt_repos": [], "common_packages": [], "nfs_service": "nfs-kernel-server", "ntp_service_name": "ntp", "packages": [], "packages_to_remove": ["multipath-tools"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "ssh"} 2021-02-11T06:05:27.703 INFO:teuthology.task.ansible.out:, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/apt_systems.yml"], "changed": false} Thursday 11 February 2021 06:05:26 +0000 (0:00:00.060) 0:01:37.275 ***** TASK [testnode : Including distro specific variables.] ************************* ok: [smithi172.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml) => {"ansible_facts": {"common_packages": ["libfcgi0ldbl", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "collectl", "blktrace", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "non_aarch64_common_packages": ["smbios-utils", "libfcgi", "sysprof"], "non_aarch64_packages_to_upgrade": ["libapache2-mod-fastcgi"], "packages_to_remove": ["openmpi-common"], "packages_to_upgrade": ["apt", "apache2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"} Thursday 11 February 2021 06:05:26 +0000 (0:00:00.084) 0:01:37.360 ***** TASK [testnode : Including major version specific variables.] ****************** ok: [smithi172.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml) => {"ansible_facts": {"non_aar 2021-02-11T06:05:27.705 INFO:teuthology.task.ansible.out:ch64_packages": ["libgoogle-perftools4", "iozone3", "libssl1.0.0"], "non_aarch64_packages_to_upgrade": [], "packages": ["mpich", "qemu-system-x86", "python-virtualenv", "python-configobj", "python-gevent", "python-numpy", "python-matplotlib", "python-nose", "btrfs-tools", "lttng-tools", "libtool-bin", "docker.io", "python3-nose"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"} Thursday 11 February 2021 06:05:26 +0000 (0:00:00.081) 0:01:37.442 ***** TASK [testnode : Including version specific variables.] ************************ ok: [smithi172.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} TASK [common : Make sure smartmontools is installed] *************************** ok: [smithi061.front.sepia.ceph.com] => {"cache_update_time": 1613023167, "cache_updated": false, "changed": false} Thursday 11 February 2021 06:05:26 +0000 (0:00:00.075) 0:01:37.517 ***** Thursday 11 February 2021 06:05:26 +0000 (0:00:00.033) 0:01:37.550 ***** TASK [testnode : Ensure the sudo group exists.] ******************************** ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} Thursday 11 February 2021 06:05:27 +0000 (0:00:00.165) 0:01:37.716 ***** TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** ok: [smithi061.front.se 2021-02-11T06:05:27.706 INFO:teuthology.task.ansible.out:pia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "size": 2716736, "state": "file", "uid": 0} TASK [testnode : Ensure the teuthology_user group exists.] ********************* ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "name": "ubuntu", "state": "present", "system": false} Thursday 11 February 2021 06:05:27 +0000 (0:00:00.168) 0:01:37.885 ***** TASK [testnode : Create the teuthology user.] ********************************** skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:05:27 +0000 (0:00:00.058) 0:01:37.944 ***** TASK [testnode : Delete the teuthology users password.] ************************ skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:05:27 +0000 (0:00:00.059) 0:01:38.003 ***** TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** ok: [smithi061.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "size": 1952038, "state": "file", "uid": 0} TASK [testnode : Add a user for xfstests to test user quotas.] ***************** ok: [smithi172.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/sh", "state": "present", "uid": 10101} Thursday 11 February 2021 06:05:27 +0000 (0:00:00.204) 0:01:38.208 ***** 2021-02-11T06:05:27.865 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-02-11T06:05:27.866 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "size": 580432, "state": "file", "uid": 0} 2021-02-11T06:05:27.902 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:27 +0000 (0:00:00.262) 0:01:38.470 ***** 2021-02-11T06:05:28.036 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "size": 4096, "state": "directory", "uid": 0} 2021-02-11T06:05:28.074 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:28 +0000 (0:00:00.172) 0:01:38.642 ***** 2021-02-11T06:05:28.081 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2021-02-11T06:05:28.083 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "size": 112, "state": "file", "uid": 0} 2021-02-11T06:05:28.119 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:28 +0000 (0:00:00.044) 0:01:38.687 ***** 2021-02-11T06:05:28.157 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* ok: [smithi172.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi172"}, "changed": false} 2021-02-11T06:05:28.192 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:28 +0000 (0:00:00.073) 0:01:38.761 ***** 2021-02-11T06:05:28.388 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-02-11T06:05:28.390 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "size": 13355, "state": "file", "uid": 0} 2021-02-11T06:05:28.662 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "size": 8602, "state": "file", "uid": 0} 2021-02-11T06:05:28.923 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "size": 3605, "state": "file", "uid": 0} 2021-02-11T06:05:28.970 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:28 +0000 (0:00:00.777) 0:01:39.538 ***** 2021-02-11T06:05:29.007 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2021-02-11T06:05:29.009 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib/nagios/plugins", "nrpe_group": "nagios", "nrpe_packages": ["nagios-nrpe-server", "nagios-plugins-basic"], "nrpe_service_name": "nagios-nrpe-server", "nrpe_user": "nagios"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/apt_systems.yml"], "changed": false} 2021-02-11T06:05:29.043 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:29 +0000 (0:00:00.073) 0:01:39.611 ***** 2021-02-11T06:05:29.050 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2021-02-11T06:05:29.050 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi172.front.sepia.ceph.com", "ansible_hostname": "smithi172", "ansible_nodename": "smithi172"}, "changed": true, "name": "smithi172"} 2021-02-11T06:05:29.086 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:29 +0000 (0:00:00.042) 0:01:39.654 ***** 2021-02-11T06:05:29.092 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2021-02-11T06:05:29.094 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:29.129 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:29 +0000 (0:00:00.042) 0:01:39.697 ***** 2021-02-11T06:05:29.155 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2021-02-11T06:05:29.156 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-11T06:05:29.190 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:29 +0000 (0:00:00.061) 0:01:39.759 ***** 2021-02-11T06:05:29.227 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2021-02-11T06:05:29.228 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-02-11T06:05:29.239 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-02-11T06:05:29.249 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-02-11T06:05:29.286 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:29 +0000 (0:00:00.095) 0:01:39.854 ***** 2021-02-11T06:05:29.313 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2021-02-11T06:05:29.315 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:29.346 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:29 +0000 (0:00:00.060) 0:01:39.914 ***** 2021-02-11T06:05:29.393 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2021-02-11T06:05:29.395 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "checksum": "2f7bfea2455a0d9220592aa0c21ac69c312e7471", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "size": 2646, "state": "file", "uid": 0} 2021-02-11T06:05:29.428 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:29 +0000 (0:00:00.081) 0:01:39.996 ***** 2021-02-11T06:05:29.729 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2021-02-11T06:05:29.731 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "size": 570, "state": "file", "uid": 0} 2021-02-11T06:05:29.783 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:29 +0000 (0:00:00.355) 0:01:40.352 ***** 2021-02-11T06:05:30.367 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2021-02-11T06:05:30.368 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-02-11T06:05:30.402 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:30 +0000 (0:00:00.618) 0:01:40.970 ***** 2021-02-11T06:05:30.431 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2021-02-11T06:05:30.433 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** ok: [smithi061.front.sepia.ceph.com] => {"cache_update_time": 1613023167, "cache_updated": false, "changed": false} 2021-02-11T06:05:30.467 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:30 +0000 (0:00:00.064) 0:01:41.035 ***** 2021-02-11T06:05:30.504 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:30 +0000 (0:00:00.037) 0:01:41.073 ***** 2021-02-11T06:05:30.533 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-02-11T06:05:30.534 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:30.570 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:30 +0000 (0:00:00.065) 0:01:41.138 ***** 2021-02-11T06:05:30.596 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-02-11T06:05:30.598 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:30.638 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:30 +0000 (0:00:00.068) 0:01:41.207 ***** 2021-02-11T06:05:30.665 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-02-11T06:05:30.667 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:30.712 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:30 +0000 (0:00:00.073) 0:01:41.281 ***** 2021-02-11T06:05:30.740 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-02-11T06:05:30.742 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:30.776 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:30 +0000 (0:00:00.063) 0:01:41.344 ***** 2021-02-11T06:05:30.781 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2021-02-11T06:05:30.782 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "checksum": "4880e187ff9f437cc33858eb1d0fa49a06d67be4", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "size": 180, "state": "file", "uid": 0} 2021-02-11T06:05:30.817 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:30 +0000 (0:00:00.040) 0:01:41.385 ***** 2021-02-11T06:05:30.823 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-02-11T06:05:30.824 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:30.877 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:30 +0000 (0:00:00.060) 0:01:41.445 ***** 2021-02-11T06:05:30.907 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-02-11T06:05:30.908 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:30.941 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:30 +0000 (0:00:00.064) 0:01:41.510 ***** 2021-02-11T06:05:30.980 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2021-02-11T06:05:30.982 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:31.015 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:31 +0000 (0:00:00.073) 0:01:41.584 ***** 2021-02-11T06:05:31.045 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2021-02-11T06:05:31.080 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:31 +0000 (0:00:00.064) 0:01:41.648 ***** 2021-02-11T06:05:31.107 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-02-11T06:05:31.142 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:31 +0000 (0:00:00.062) 0:01:41.710 ***** 2021-02-11T06:05:31.150 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2021-02-11T06:05:31.151 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/nagios/plugins/check_mem.sh", "size": 1964, "state": "file", "uid": 0} 2021-02-11T06:05:31.192 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:31 +0000 (0:00:00.049) 0:01:41.760 ***** 2021-02-11T06:05:31.201 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2021-02-11T06:05:31.241 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:31 +0000 (0:00:00.048) 0:01:41.809 ***** 2021-02-11T06:05:31.269 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2021-02-11T06:05:31.270 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:31.308 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:31 +0000 (0:00:00.067) 0:01:41.876 ***** 2021-02-11T06:05:31.338 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-02-11T06:05:31.339 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "DAEMON_OPTS", "msg": ""} 2021-02-11T06:05:31.350 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2021-02-11T06:05:31.351 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-02-11T06:05:31.386 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:31 +0000 (0:00:00.077) 0:01:41.954 ***** 2021-02-11T06:05:31.412 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2021-02-11T06:05:31.413 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:31.452 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:31 +0000 (0:00:00.066) 0:01:42.021 ***** 2021-02-11T06:05:31.462 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-02-11T06:05:31.462 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "NRPE_OPTS", "msg": ""} 2021-02-11T06:05:31.507 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:31 +0000 (0:00:00.054) 0:01:42.076 ***** 2021-02-11T06:05:31.514 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-11T06:05:31.515 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:31.551 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:31 +0000 (0:00:00.043) 0:01:42.119 ***** 2021-02-11T06:05:31.557 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2021-02-11T06:05:31.558 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:31.593 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:31 +0000 (0:00:00.042) 0:01:42.161 ***** 2021-02-11T06:05:31.600 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2021-02-11T06:05:31.602 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:31.637 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:31 +0000 (0:00:00.044) 0:01:42.206 ***** 2021-02-11T06:05:31.644 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2021-02-11T06:05:31.645 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-11T06:05:31.680 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:31 +0000 (0:00:00.042) 0:01:42.248 ***** 2021-02-11T06:05:31.695 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2021-02-11T06:05:31.696 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:31.731 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:31 +0000 (0:00:00.051) 0:01:42.299 ***** 2021-02-11T06:05:31.738 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2021-02-11T06:05:31.740 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:31.776 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:31 +0000 (0:00:00.044) 0:01:42.344 ***** 2021-02-11T06:05:31.782 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2021-02-11T06:05:31.783 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:31.820 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:31 +0000 (0:00:00.044) 0:01:42.389 ***** 2021-02-11T06:05:31.846 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2021-02-11T06:05:31.848 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:31.889 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:31 +0000 (0:00:00.068) 0:01:42.458 ***** 2021-02-11T06:05:31.917 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-02-11T06:05:31.918 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:31.955 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:31 +0000 (0:00:00.065) 0:01:42.523 ***** 2021-02-11T06:05:31.993 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-02-11T06:05:31.995 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:32.032 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:32 +0000 (0:00:00.076) 0:01:42.600 ***** 2021-02-11T06:05:32.069 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-02-11T06:05:32.070 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:32.112 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:32 +0000 (0:00:00.080) 0:01:42.680 ***** 2021-02-11T06:05:32.118 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2021-02-11T06:05:32.119 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "checksum": "a0ee4c31e844cf26108115f6af99f92f904eafbf", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "size": 1176, "state": "file", "uid": 0} 2021-02-11T06:05:32.169 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:32 +0000 (0:00:00.056) 0:01:42.737 ***** 2021-02-11T06:05:32.177 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2021-02-11T06:05:32.178 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:32.214 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:32 +0000 (0:00:00.045) 0:01:42.782 ***** 2021-02-11T06:05:32.242 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-02-11T06:05:32.244 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:32.275 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:32 +0000 (0:00:00.061) 0:01:42.844 ***** 2021-02-11T06:05:32.302 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-02-11T06:05:32.303 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:32.337 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:32 +0000 (0:00:00.061) 0:01:42.905 ***** 2021-02-11T06:05:32.364 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2021-02-11T06:05:32.365 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:32.399 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:32 +0000 (0:00:00.062) 0:01:42.967 ***** 2021-02-11T06:05:32.406 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2021-02-11T06:05:32.407 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nagios-nrpe-server", "state": "started", "status": {"ActiveEnterTimestamp": "Thu 2021-02-11 06:03:10 UTC", "ActiveEnterTimestampMonotonic": "19945197", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "network.target nss-lookup.target system.slice systemd-journald.socket remote-fs.target basic.target sysinit.target time-sync.target systemd-tmpfiles-setup.service local-fs.target -.mount var-run.mount", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2021-02-11 06:03:10 UTC", "AssertTimestampMonotonic": "19944271", "Before": "getty@tty1.service xdm.service shutdown.target multi-user.target plymouth-quit.service", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2021-02-11 06:03:10 UTC", "ConditionTimestampMonotonic": "19944270", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nagios-nrpe-ser 2021-02-11T06:05:32.409 INFO:teuthology.task.ansible.out:ver.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EnvironmentFile": "/etc/default/nagios-nrpe-server (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1291", "ExecMainStartTimestamp": "Thu 2021-02-11 06:03:10 UTC", "ExecMainStartTimestampMonotonic": "19945163", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; ignore_errors=no ; start_time=[Thu 2021-02-11 06:03:10 UTC] ; stop_time=[n/a] ; pid=1291 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nagios/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/nagios-nrpe-server.service", "GID": "118", "Group": "nagios", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nagios-nrpe-server.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2021-02-11 06:03:10 UTC", "InactiveExitTimestampMonotonic": "19945197", "InvocationID": "77c5992e68c34c9ca16f4277d738d34b", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "i 2021-02-11T06:05:32.410 INFO:teuthology.task.ansible.out:nfinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128029", "LimitNPROCSoft": "128029", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "128029", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1291", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nagios-nrpe-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nagios/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount sysinit.target system.slice", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", 2021-02-11T06:05:32.411 INFO:teuthology.task.ansible.out: "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2021-02-11 06:03:10 UTC", "StateChangeTimestampMonotonic": "19945197", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "109", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "User": "nagios", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2021-02-11 06:03:10 UTC", "WatchdogTimestampMonotonic": "19945196", "WatchdogUSec": "0"}} 2021-02-11T06:05:32.445 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:32 +0000 (0:00:00.045) 0:01:43.013 ***** 2021-02-11T06:05:32.452 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2021-02-11T06:05:32.453 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:32.492 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:32 +0000 (0:00:00.046) 0:01:43.060 ***** 2021-02-11T06:05:32.498 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2021-02-11T06:05:32.498 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:32.534 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:32 +0000 (0:00:00.042) 0:01:43.102 ***** 2021-02-11T06:05:32.540 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2021-02-11T06:05:32.542 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:32.577 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:32 +0000 (0:00:00.043) 0:01:43.145 ***** 2021-02-11T06:05:32.583 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2021-02-11T06:05:32.584 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:32.619 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:32 +0000 (0:00:00.042) 0:01:43.188 ***** 2021-02-11T06:05:32.624 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2021-02-11T06:05:32.626 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:32.660 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:32 +0000 (0:00:00.040) 0:01:43.228 ***** 2021-02-11T06:05:32.665 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2021-02-11T06:05:32.665 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:32.701 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:32 +0000 (0:00:00.040) 0:01:43.269 ***** 2021-02-11T06:05:32.707 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2021-02-11T06:05:32.708 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:32.743 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:32 +0000 (0:00:00.042) 0:01:43.312 ***** 2021-02-11T06:05:32.749 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2021-02-11T06:05:32.750 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:32.785 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:32 +0000 (0:00:00.041) 0:01:43.353 ***** 2021-02-11T06:05:32.794 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2021-02-11T06:05:32.795 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:32.829 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:32 +0000 (0:00:00.044) 0:01:43.397 ***** 2021-02-11T06:05:32.835 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2021-02-11T06:05:32.837 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:32.872 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:32 +0000 (0:00:00.042) 0:01:43.440 ***** 2021-02-11T06:05:32.878 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2021-02-11T06:05:32.879 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:32.914 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:32 +0000 (0:00:00.042) 0:01:43.482 ***** 2021-02-11T06:05:32.920 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2021-02-11T06:05:32.922 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:32.957 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:32 +0000 (0:00:00.042) 0:01:43.525 ***** 2021-02-11T06:05:32.963 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2021-02-11T06:05:32.965 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:33.001 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:32 +0000 (0:00:00.044) 0:01:43.569 ***** 2021-02-11T06:05:33.008 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2021-02-11T06:05:33.010 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:33.045 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:33 +0000 (0:00:00.044) 0:01:43.614 ***** 2021-02-11T06:05:35.776 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:05:33 +0000 (0:00:00.062) 0:01:43.676 ***** TASK [testnode : Check for custom repos] *************************************** ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "cmd": "ls -1 /etc/apt/sources.list.d/", "delta": "0:00:00.005268", "end": "2021-02-11 06:05:33.109703", "rc": 0, "start": "2021-02-11 06:05:33.104435", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} TASK [common : nrpe - Load SELinux policy package] ***************************** skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:05:33 +0000 (0:00:00.068) 0:01:43.745 ***** Thursday 11 February 2021 06:05:33 +0000 (0:00:00.033) 0:01:43.778 ***** TASK [common : nrpe - Remove temporary files] ********************************** skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:05:33 +0000 (0:00:00.042) 0:01:43.821 ***** TASK [testnode : Remove custom repos] ****************************************** Thursday 11 February 2021 06:05:33 +0000 (0:00:00.046) 0:01:43.867 ***** TASK [testnode : Include package type specific vars.] ************************** ok: [smithi061.front.sepia.ceph.com] => {"ansible_facts": {"apt_repos": [], "ceph_packages_to_remove": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common"], "common_apt_repos": [], "common_packages": [], "nfs_service": "nfs-kernel-server", "ntp_service_name": "ntp", "packages": [], "packages_to_remove": ["multipath-tools"], "pip_packages_to_i 2021-02-11T06:05:35.777 INFO:teuthology.task.ansible.out:nstall": ["remoto>=0.0.35"], "ssh_service_name": "ssh"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/apt_systems.yml"], "changed": false} Thursday 11 February 2021 06:05:33 +0000 (0:00:00.044) 0:01:43.912 ***** TASK [testnode : Including distro specific variables.] ************************* ok: [smithi061.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml) => {"ansible_facts": {"common_packages": ["libfcgi0ldbl", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "collectl", "blktrace", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "non_aarch64_common_packages": ["smbios-utils", "libfcgi", "sysprof"], "non_aarch64_packages_to_upgrade": ["libapache2-mod-fastcgi"], "packages_to_remove": ["openmpi-common"], "packages_to_upgrade": ["apt", "apache2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"} Thursday 11 February 2021 06:05:33 +0000 (0:00:00.075) 0:01:43.988 ***** TASK [testnode : Including major version specific variables.] ****************** ok: [smithi061.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/test 2021-02-11T06:05:35.778 INFO:teuthology.task.ansible.out:node/vars/ubuntu_18.yml) => {"ansible_facts": {"non_aarch64_packages": ["libgoogle-perftools4", "iozone3", "libssl1.0.0"], "non_aarch64_packages_to_upgrade": [], "packages": ["mpich", "qemu-system-x86", "python-virtualenv", "python-configobj", "python-gevent", "python-numpy", "python-matplotlib", "python-nose", "btrfs-tools", "lttng-tools", "libtool-bin", "docker.io", "python3-nose"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"} Thursday 11 February 2021 06:05:33 +0000 (0:00:00.076) 0:01:44.064 ***** TASK [testnode : Including version specific variables.] ************************ ok: [smithi061.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} Thursday 11 February 2021 06:05:33 +0000 (0:00:00.071) 0:01:44.136 ***** TASK [testnode : Set apt preferences] ****************************************** ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "checksum": "ba101482124806c7f6ea9fc7b4acfda303391ad6", "dest": "/etc/apt/preferences.d/ceph.pref", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/apt/preferences.d/ceph.pref", "size": 52, "state": "file", "uid": 0} Thursday 11 February 2021 06:05:33 +0000 (0:00:00.083) 0:01:44.220 ***** TASK [testnode : Add sources list] ********************************************* skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Con 2021-02-11T06:05:35.780 INFO:teuthology.task.ansible.out:ditional result was False"} Thursday 11 February 2021 06:05:33 +0000 (0:00:00.059) 0:01:44.279 ***** TASK [testnode : Ensure the sudo group exists.] ******************************** ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} Thursday 11 February 2021 06:05:33 +0000 (0:00:00.046) 0:01:44.326 ***** TASK [testnode : Ensure the teuthology_user group exists.] ********************* ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "name": "ubuntu", "state": "present", "system": false} Thursday 11 February 2021 06:05:33 +0000 (0:00:00.172) 0:01:44.498 ***** TASK [testnode : Create the teuthology user.] ********************************** skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:05:34 +0000 (0:00:00.076) 0:01:44.575 ***** TASK [testnode : Delete the teuthology users password.] ************************ skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:05:34 +0000 (0:00:00.061) 0:01:44.637 ***** TASK [testnode : Add a user for xfstests to test user quotas.] ***************** ok: [smithi061.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/sh", "state": "present", "uid": 10101} Thursday 11 February 2021 06:05:34 +0000 (0:00:00.184) 0:01:44.821 ***** TASK [testnode : Set a high max open files limit for the teuthology user.] ***** ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "size": 2021-02-11T06:05:35.781 INFO:teuthology.task.ansible.out:112, "state": "file", "uid": 0} Thursday 11 February 2021 06:05:34 +0000 (0:00:00.394) 0:01:45.215 ***** TASK [testnode : Set hostname var] ********************************************* ok: [smithi061.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi061"}, "changed": false} Thursday 11 February 2021 06:05:34 +0000 (0:00:00.060) 0:01:45.276 ***** TASK [testnode : Install apt keys] ********************************************* ok: [smithi172.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc"} TASK [testnode : Set the system's hostname] ************************************ changed: [smithi061.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi061.front.sepia.ceph.com", "ansible_hostname": "smithi061", "ansible_nodename": "smithi061"}, "changed": true, "name": "smithi061"} Thursday 11 February 2021 06:05:35 +0000 (0:00:00.441) 0:01:45.717 ***** TASK [testnode : Upload distro major version specific sshd_config] ************* ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "checksum": "2f7bfea2455a0d9220592aa0c21ac69c312e7471", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "size": 2646, "state": "file", "uid": 0} Thursday 11 February 2021 06:05:35 +0000 (0:00:00.351) 0:01:46.068 ***** 2021-02-11T06:05:35.786 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2021-02-11T06:05:35.786 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "size": 570, "state": "file", "uid": 0} 2021-02-11T06:05:35.822 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:35 +0000 (0:00:00.322) 0:01:46.390 ***** 2021-02-11T06:05:35.916 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2021-02-11T06:05:35.917 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc"} 2021-02-11T06:05:35.951 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:35 +0000 (0:00:00.129) 0:01:46.520 ***** 2021-02-11T06:05:36.366 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2021-02-11T06:05:36.368 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-02-11T06:05:36.403 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:36 +0000 (0:00:00.452) 0:01:46.972 ***** 2021-02-11T06:05:36.431 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2021-02-11T06:05:36.432 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:36.465 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:36 +0000 (0:00:00.061) 0:01:47.033 ***** 2021-02-11T06:05:36.492 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-02-11T06:05:36.493 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:36.527 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:36 +0000 (0:00:00.061) 0:01:47.095 ***** 2021-02-11T06:05:36.554 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-02-11T06:05:36.556 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:36.589 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:36 +0000 (0:00:00.062) 0:01:47.158 ***** 2021-02-11T06:05:36.617 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:36.653 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:36 +0000 (0:00:00.063) 0:01:47.221 ***** 2021-02-11T06:05:36.680 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-02-11T06:05:36.681 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:36.713 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:36 +0000 (0:00:00.060) 0:01:47.282 ***** 2021-02-11T06:05:36.740 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-02-11T06:05:36.741 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:36.773 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:36 +0000 (0:00:00.060) 0:01:47.342 ***** 2021-02-11T06:05:36.800 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:36.833 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:36 +0000 (0:00:00.059) 0:01:47.401 ***** 2021-02-11T06:05:36.860 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2021-02-11T06:05:36.861 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"cache_update_time": 1613023222, "cache_updated": false, "changed": false} TASK [testnode : Configure local mirrorlists] ********************************** skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:36.894 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:36 +0000 (0:00:00.061) 0:01:47.463 ***** 2021-02-11T06:05:36.933 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:36 +0000 (0:00:00.038) 0:01:47.501 ***** 2021-02-11T06:05:36.940 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2021-02-11T06:05:36.980 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:36 +0000 (0:00:00.047) 0:01:47.549 ***** 2021-02-11T06:05:50.387 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** Thursday 11 February 2021 06:05:37 +0000 (0:00:00.043) 0:01:47.592 ***** TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** Thursday 11 February 2021 06:05:37 +0000 (0:00:00.046) 0:01:47.639 ***** TASK [testnode : Enable copr repos] ******************************************** Thursday 11 February 2021 06:05:37 +0000 (0:00:00.064) 0:01:47.703 ***** TASK [testnode : Install GPG] ************************************************** skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:05:37 +0000 (0:00:00.062) 0:01:47.765 ***** TASK [testnode : Blast the apt cache.] ***************************************** [WARNING]: Consider using 'become', 'become_method', and 'become_user' rather than running sudo changed: [smithi172.front.sepia.ceph.com] => {"changed": true, "cmd": ["sudo", "apt-get", "clean"], "delta": "0:00:00.033915", "end": "2021-02-11 06:05:37.165747", "rc": 0, "start": "2021-02-11 06:05:37.131832", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} Thursday 11 February 2021 06:05:37 +0000 (0:00:00.043) 0:01:47.808 ***** TASK [testnode : Install GPG keys] ********************************************* skipping: [smithi061.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:05:37 +0000 (0:00:00.044) 0:01:47.853 ***** TASK [te 2021-02-11T06:05:50.388 INFO:teuthology.task.ansible.out:stnode : Clean yum cache] ********************************************** skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:05:37 +0000 (0:00:00.061) 0:01:47.915 ***** TASK [testnode : set_fact] ***************************************************** skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:05:37 +0000 (0:00:00.063) 0:01:47.979 ***** TASK [testnode : Ensure yum_utils is present.] ********************************* skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:05:37 +0000 (0:00:00.063) 0:01:48.042 ***** TASK [testnode : Removing saved yum transactions] ****************************** skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:05:37 +0000 (0:00:00.062) 0:01:48.105 ***** TASK [testnode : Check if ceph-debuginfo is installed] ************************* skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:05:37 +0000 (0:00:00.062) 0:01:48.168 ***** TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:05:37 +0000 (0:00:00.062) 0:01:48.230 ***** TASK [testnode : Ensure ceph packages are not present.] ************************ skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:05:37 +0000 (0:00:00.066) 0:01:48.296 ***** TASK [testnode : Ensure ceph dependency packages a 2021-02-11T06:05:50.390 INFO:teuthology.task.ansible.out:re not present.] ************* skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:05:37 +0000 (0:00:00.060) 0:01:48.357 ***** TASK [testnode : Install packages] ********************************************* skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:05:37 +0000 (0:00:00.059) 0:01:48.417 ***** TASK [testnode : Install epel packages] **************************************** skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:05:37 +0000 (0:00:00.061) 0:01:48.478 ***** TASK [testnode : Remove packages] ********************************************** skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:05:37 +0000 (0:00:00.064) 0:01:48.542 ***** TASK [testnode : Upgrade packages] ********************************************* skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:05:38 +0000 (0:00:00.074) 0:01:48.616 ***** TASK [testnode : Stop and disable firewalld] *********************************** skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:05:38 +0000 (0:00:00.062) 0:01:48.679 ***** TASK [testnode : Stop and disable iptables] ************************************ skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:05:38 +0000 (0:00:00.062) 0:01:48.741 ***** TASK [testnode : Enable SELinux] *********************************************** skipping: [smit 2021-02-11T06:05:50.391 INFO:teuthology.task.ansible.out:hi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:05:38 +0000 (0:00:00.062) 0:01:48.803 ***** TASK [testnode : Install abrt] ************************************************* skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:05:38 +0000 (0:00:00.084) 0:01:48.887 ***** TASK [testnode : Enable abrt-auto-reporting] *********************************** skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:05:38 +0000 (0:00:00.061) 0:01:48.949 ***** TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:05:38 +0000 (0:00:00.059) 0:01:49.009 ***** TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:05:38 +0000 (0:00:00.063) 0:01:49.072 ***** TASK [testnode : Restart abrtd] ************************************************ skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:05:38 +0000 (0:00:00.063) 0:01:49.136 ***** TASK [testnode : Check for custom repos] *************************************** ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "cmd": "ls -1 /etc/apt/sources.list.d/", "delta": "0:00:00.005316", "end": "2021-02-11 06:05:38.480791", "rc": 0, "start": "2021-02-11 06:05:38.475475", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} Thursday 11 February 2021 06:05:38 +0000 (0:00:00.168) 2021-02-11T06:05:50.392 INFO:teuthology.task.ansible.out: 0:01:49.305 ***** TASK [testnode : Remove custom repos] ****************************************** Thursday 11 February 2021 06:05:38 +0000 (0:00:00.060) 0:01:49.366 ***** TASK [testnode : Set apt preferences] ****************************************** ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "checksum": "ba101482124806c7f6ea9fc7b4acfda303391ad6", "dest": "/etc/apt/preferences.d/ceph.pref", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/apt/preferences.d/ceph.pref", "size": 52, "state": "file", "uid": 0} Thursday 11 February 2021 06:05:39 +0000 (0:00:00.344) 0:01:49.710 ***** TASK [testnode : Add sources list] ********************************************* skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:05:39 +0000 (0:00:00.061) 0:01:49.772 ***** TASK [testnode : Update apt cache.] ******************************************** ok: [smithi172.front.sepia.ceph.com] => {"cache_update_time": 1613023222, "cache_updated": false, "changed": false} Thursday 11 February 2021 06:05:40 +0000 (0:00:01.015) 0:01:50.788 ***** TASK [testnode : Install apt keys] ********************************************* ok: [smithi061.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc"} ok: [smithi061.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc"} Thursday 11 February 2021 06:05:41 +0000 (0:00:00.965) 0:01:51.753 ***** TASK [testnode : Install python-apt] ******************************************* ok: [smithi061.front.sepia.ceph.com] => {"cache_update_time": 1613023167, "cache_updated": false, "changed": false} Thursday 11 February 20 2021-02-11T06:05:50.394 INFO:teuthology.task.ansible.out:21 06:05:42 +0000 (0:00:00.972) 0:01:52.726 ***** TASK [testnode : Add local apt repos.] ***************************************** Thursday 11 February 2021 06:05:42 +0000 (0:00:00.060) 0:01:52.786 ***** TASK [testnode : Blast the apt cache.] ***************************************** changed: [smithi061.front.sepia.ceph.com] => {"changed": true, "cmd": ["sudo", "apt-get", "clean"], "delta": "0:00:00.035517", "end": "2021-02-11 06:05:42.162486", "rc": 0, "start": "2021-02-11 06:05:42.126969", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} Thursday 11 February 2021 06:05:42 +0000 (0:00:00.198) 0:01:52.984 ***** TASK [testnode : Update apt cache.] ******************************************** ok: [smithi061.front.sepia.ceph.com] => {"cache_update_time": 1613023167, "cache_updated": false, "changed": false} Thursday 11 February 2021 06:05:45 +0000 (0:00:03.118) 0:01:56.103 ***** TASK [testnode : Ensure packages are not present.] ***************************** changed: [smithi172.front.sepia.ceph.com] => {"changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirpa 2021-02-11T06:05:50.395 INFO:teuthology.task.ansible.out:rser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following packages will be REMOVED:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 0 newly installed, 6 to remove and 19 not upgraded.\nAfter this operation, 47.6 MB disk space will be freed.\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137610 files and directories currently installed.)\r\nRemoving qemu-utils (1:2.11+dfsg-1ubuntu7.34) ...\r\nRemoving qemu-system-x86 (1:2.11+dfsg-1ubuntu7.34) ...\r\nRemoving qemu-system-common (1:2.11+dfsg-1ubuntu7.34) ...\r\nRemoving qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.34) ...\r\nRemoving librbd1 (12.2.13-0ubuntu0.18.04.6) ...\r\nRemoving librados2 (12.2.13-0ubuntu0.18.04.6) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1.4) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1. 2021-02-11T06:05:50.396 INFO:teuthology.task.ansible.out:0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following packages will be REMOVED:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 0 newly installed, 6 to remove and 19 not upgraded.", "After this operation, 47.6 MB disk space will be freed.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137610 files and directories currently installed.)", "Removing qemu-utils (1:2.11+dfsg-1ubuntu7.34) ...", "Removing qemu-system-x86 (1:2.11+dfsg-1ubuntu7.34) ...", "Removing qemu-system-common (1:2.11+dfsg-1ubuntu7.34) ...", "Removing qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.34) ...", "Removing librbd1 (12.2.13-0ubuntu0.18.04.6) ...", "Removing librados2 (12.2.13-0ubuntu0.18.04.6) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.4) ..."]} Thursday 11 February 2021 06:05:49 +0000 (0:00:04.207) 0:02:00.310 ***** 2021-02-11T06:05:50.635 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-02-11T06:05:50.636 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"cache_update_time": 1613023222, "cache_updated": false, "changed": false} 2021-02-11T06:05:50.671 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:50 +0000 (0:00:00.929) 0:02:01.240 ***** 2021-02-11T06:05:50.698 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2021-02-11T06:05:50.700 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:05:50.734 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:50 +0000 (0:00:00.062) 0:02:01.303 ***** 2021-02-11T06:05:54.499 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2021-02-11T06:05:54.749 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => {"changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following packages will be REMOVED:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 0 newly installed, 6 to remove and 19 not upgraded.\nAfter this operation, 47.6 MB disk space will be freed.\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137610 files and directorie 2021-02-11T06:05:54.750 INFO:teuthology.task.ansible.out:s currently installed.)\r\nRemoving qemu-utils (1:2.11+dfsg-1ubuntu7.34) ...\r\nRemoving qemu-system-x86 (1:2.11+dfsg-1ubuntu7.34) ...\r\nRemoving qemu-system-common (1:2.11+dfsg-1ubuntu7.34) ...\r\nRemoving qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.34) ...\r\nRemoving librbd1 (12.2.13-0ubuntu0.18.04.6) ...\r\nRemoving librados2 (12.2.13-0ubuntu0.18.04.6) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1.4) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following packages will be REMOVED:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 0 newly installed, 6 to remove and 19 not upgraded.", "After this operation, 47.6 MB disk space will be freed.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Read 2021-02-11T06:05:54.752 INFO:teuthology.task.ansible.out:ing database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137610 files and directories currently installed.)", "Removing qemu-utils (1:2.11+dfsg-1ubuntu7.34) ...", "Removing qemu-system-x86 (1:2.11+dfsg-1ubuntu7.34) ...", "Removing qemu-system-common (1:2.11+dfsg-1ubuntu7.34) ...", "Removing qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.34) ...", "Removing librbd1 (12.2.13-0ubuntu0.18.04.6) ...", "Removing librados2 (12.2.13-0ubuntu0.18.04.6) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.4) ..."]} Thursday 11 February 2021 06:05:54 +0000 (0:00:03.799) 0:02:05.102 ***** 2021-02-11T06:05:55.436 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-02-11T06:05:56.411 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"cache_update_time": 1613023167, "cache_updated": false, "changed": false} Thursday 11 February 2021 06:05:55 +0000 (0:00:00.946) 0:02:06.048 ***** TASK [testnode : Upgrade non aarch64 packages] ********************************* skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:05:55 +0000 (0:00:00.067) 0:02:06.116 ***** 2021-02-11T06:05:59.387 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-02-11T06:05:59.388 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => {"cache_update_time": 1613023222, "cache_updated": false, "changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\ngit is already the newest version (1:2.17.1-1ubuntu0.7).\nThe following additional packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils\nSuggested packages:\n samba vde2 sgabios ovmf debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 6 newly installed, 0 to remove and 19 not upgraded.\nNeed to get 10.4 MB of archives.\nAfter this operation, 47.6 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.6 [2724 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.6 [925 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.35 [40.5 kB]\nGet:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.35 [672 kB]\nGet:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.35 [5194 kB]\nGet:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.35 [868 kB]\nFetched 10.4 MB in 1s (9740 kB/s)\nSelecting previously unselected package librados2.\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database .. 2021-02-11T06:05:59.389 INFO:teuthology.task.ansible.out:. 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137468 files and directories currently installed.)\r\nPreparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.6_amd64.deb ...\r\nUnpacking librados2 (12.2.13-0ubuntu0.18.04.6) ...\r\nSelecting previously unselected package librbd1.\r\nPreparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.6_amd64.deb ...\r\nUnpacking librbd1 (12.2.13-0ubuntu0.18.04.6) ...\r\nSelecting previously unselected package qemu-block-extra:amd64.\r\nPreparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.35_amd64.deb ...\r\nUnpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.35) ...\r\nSelecting previously unselected package qemu-system-common.\r\nPreparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.35_amd64.deb ...\r\nUnpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.35) ...\r\nSelecting previously unselected package qemu-system-x86.\r\nPreparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.35_amd64.deb ...\r\nUnpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.35) ...\r\nSelecting previously unselected package qemu-utils.\r\nPreparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.35_amd64.deb ...\r\nUnpacking qemu-utils (1:2.11+dfsg-1ubuntu7.35) ...\r\nSetting up librados2 (12.2.13-0ubuntu0.18.04.6) ...\r\nSetting up librbd1 (12.2.13-0ubuntu0.18.04.6) ...\r\nSetting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.35) ...\r\nSetting up qemu-utils (1:2.11+dfsg-1ubuntu7.35) ...\r\nSetting up qemu-system-common (1:2.11+dfsg-1ubuntu7.35) ...\r\nSetting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.35) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubunt 2021-02-11T06:05:59.391 INFO:teuthology.task.ansible.out:u1.4) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "git is already the newest version (1:2.17.1-1ubuntu0.7).", "The following additional packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils", "Suggested packages:", " samba vde2 sgabios ovmf debootstrap", "The following NEW packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 6 newly installed, 0 to remove and 19 not upgraded.", "Need to get 10.4 MB of archives.", "After this operation, 47.6 MB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.6 [2724 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.6 [925 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.35 [40.5 kB]", "Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.35 [672 kB]", "Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.35 [5194 kB]", "Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.35 [868 kB]", "Fetched 10.4 MB in 1s (9740 kB/s)", "Selecting previously unselected package librados2.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(R 2021-02-11T06:05:59.392 INFO:teuthology.task.ansible.out:eading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137468 files and directories currently installed.)", "Preparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.6_amd64.deb ...", "Unpacking librados2 (12.2.13-0ubuntu0.18.04.6) ...", "Selecting previously unselected package librbd1.", "Preparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.6_amd64.deb ...", "Unpacking librbd1 (12.2.13-0ubuntu0.18.04.6) ...", "Selecting previously unselected package qemu-block-extra:amd64.", "Preparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.35_amd64.deb ...", "Unpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.35) ...", "Selecting previously unselected package qemu-system-common.", "Preparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.35_amd64.deb ...", "Unpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.35) ...", "Selecting previously unselected package qemu-system-x86.", "Preparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.35_amd64.deb ...", "Unpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.35) ...", "Selecting previously unselected package qemu-utils.", "Preparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.35_amd64.deb ...", "Unpacking qemu-utils (1:2.11+dfsg-1ubuntu7.35) ...", "Setting up librados2 (12.2.13-0ubuntu0.18.04.6) ...", "Setting up librbd1 (12.2.13-0ubuntu0.18.04.6) ...", "Setting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.35) ...", "Setting up qemu-utils (1:2.11+dfsg-1ubuntu7.35) ...", "Setting up qemu-system-common (1:2.11+dfsg-1ubuntu7.35) ...", "Setting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.35) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.4) ..."]} 2021-02-11T06:05:59.428 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:05:59 +0000 (0:00:03.880) 0:02:09.996 ***** 2021-02-11T06:06:00.276 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2021-02-11T06:06:00.277 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"cache_update_time": 1613023222, "cache_updated": false, "changed": false} 2021-02-11T06:06:00.312 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:00 +0000 (0:00:00.884) 0:02:10.881 ***** 2021-02-11T06:06:00.444 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2021-02-11T06:06:00.445 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "cmd": ["perl", "-pe", "if (m{^([^#]\\S*\\s+/\\s+\\S+\\s+)(\\S+)(\\s+.*)$}) { $_=\"$1$2,user_xattr$3\\n\" unless $2=~m{(^|,)user_xattr(,|$)}; }", "-i.bak", "/etc/fstab"], "rc": 0, "stdout": "skipped, since /etc/fstab.bak exists", "stdout_lines": ["skipped, since /etc/fstab.bak exists"]} 2021-02-11T06:06:00.479 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:00 +0000 (0:00:00.166) 0:02:11.047 ***** 2021-02-11T06:06:00.505 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2021-02-11T06:06:00.506 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:00.540 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:00 +0000 (0:00:00.061) 0:02:11.109 ***** 2021-02-11T06:06:00.670 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2021-02-11T06:06:00.672 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "fuse", "state": "present", "system": false} 2021-02-11T06:06:00.705 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:00 +0000 (0:00:00.164) 0:02:11.273 ***** 2021-02-11T06:06:01.012 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2021-02-11T06:06:01.013 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "checksum": "9edff47031d0f14a9b8c813b44eb4a9296d0439f", "dest": "/etc/fuse.conf", "gid": 10102, "group": "fuse", "mode": "0644", "owner": "root", "path": "/etc/fuse.conf", "size": 366, "state": "file", "uid": 0} 2021-02-11T06:06:01.049 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:01 +0000 (0:00:00.343) 0:02:11.617 ***** 2021-02-11T06:06:01.221 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2021-02-11T06:06:01.223 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1238, "groups": "fuse,kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-02-11T06:06:01.256 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:01 +0000 (0:00:00.207) 0:02:11.824 ***** 2021-02-11T06:06:01.386 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2021-02-11T06:06:01.387 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-11T06:06:01.423 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:01 +0000 (0:00:00.167) 0:02:11.992 ***** 2021-02-11T06:06:01.710 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2021-02-11T06:06:01.711 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => {"changed": true, "name": "apache2", "state": "stopped", "status": {"ActiveEnterTimestamp": "Thu 2021-02-11 06:03:21 UTC", "ActiveEnterTimestampMonotonic": "19134195", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "nss-lookup.target systemd-tmpfiles-setup.service remote-fs.target system.slice sysinit.target network.target basic.target -.mount systemd-journald.socket", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2021-02-11 06:03:21 UTC", "AssertTimestampMonotonic": "18752314", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2021-02-11 06:03:21 UTC", "ConditionTimestampMonotonic": "18752314", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/apache2.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "The Apache HTTP Server", "DevicePolicy": "auto", " 2021-02-11T06:06:01.712 INFO:teuthology.task.ansible.out:DropInPaths": "/lib/systemd/system/apache2.service.d/apache2-systemd.conf", "DynamicUser": "no", "Environment": "APACHE_STARTED_BY_SYSTEMD=true", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1371", "ExecMainStartTimestamp": "Thu 2021-02-11 06:03:21 UTC", "ExecMainStartTimestampMonotonic": "19134176", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/apache2.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "apache2.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2021-02-11 06:03:21 UTC", "InactiveExitTimestampMonotonic": "18752977", "InvocationID": "ec0a86bf68a34f0c96c34032abe09f93", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESof 2021-02-11T06:06:01.713 INFO:teuthology.task.ansible.out:t": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128029", "LimitNPROCSoft": "128029", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "128029", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1371", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "apache2.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice -.mount sysinit.target", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIG 2021-02-11T06:06:01.715 INFO:teuthology.task.ansible.out:HUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2021-02-11 06:03:21 UTC", "StateChangeTimestampMonotonic": "19134195", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "55", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2021-02-11 06:03:21 UTC", "WatchdogTimestampMonotonic": "19134193", "WatchdogUSec": "0"}} 2021-02-11T06:06:01.745 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:01 +0000 (0:00:00.321) 0:02:12.314 ***** 2021-02-11T06:06:01.772 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-02-11T06:06:01.773 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:01.805 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:01 +0000 (0:00:00.059) 0:02:12.373 ***** 2021-02-11T06:06:01.830 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-02-11T06:06:01.831 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:01.863 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:01 +0000 (0:00:00.057) 0:02:12.431 ***** 2021-02-11T06:06:01.888 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-02-11T06:06:01.890 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:01.922 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:01 +0000 (0:00:00.059) 0:02:12.490 ***** 2021-02-11T06:06:01.948 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-02-11T06:06:01.950 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:01.980 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:01 +0000 (0:00:00.058) 0:02:12.549 ***** 2021-02-11T06:06:02.006 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-02-11T06:06:02.007 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:02.039 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:02 +0000 (0:00:00.058) 0:02:12.607 ***** 2021-02-11T06:06:02.064 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-02-11T06:06:02.066 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:02.096 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:02 +0000 (0:00:00.057) 0:02:12.665 ***** 2021-02-11T06:06:02.121 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-02-11T06:06:02.123 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:02.155 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:02 +0000 (0:00:00.058) 0:02:12.723 ***** 2021-02-11T06:06:02.181 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-02-11T06:06:02.183 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:02.215 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:02 +0000 (0:00:00.059) 0:02:12.784 ***** 2021-02-11T06:06:02.242 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-11T06:06:02.244 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:02.275 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:02 +0000 (0:00:00.059) 0:02:12.844 ***** 2021-02-11T06:06:02.301 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-02-11T06:06:02.302 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:02.338 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:02 +0000 (0:00:00.062) 0:02:12.906 ***** 2021-02-11T06:06:02.365 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-02-11T06:06:02.366 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:02.398 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:02 +0000 (0:00:00.059) 0:02:12.966 ***** 2021-02-11T06:06:02.425 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-02-11T06:06:02.426 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:02.457 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:02 +0000 (0:00:00.059) 0:02:13.026 ***** 2021-02-11T06:06:02.483 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-02-11T06:06:02.484 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:02.516 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:02 +0000 (0:00:00.058) 0:02:13.084 ***** 2021-02-11T06:06:02.541 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-02-11T06:06:02.543 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:02.574 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:02 +0000 (0:00:00.058) 0:02:13.143 ***** 2021-02-11T06:06:02.601 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-02-11T06:06:02.602 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:02.635 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:02 +0000 (0:00:00.060) 0:02:13.203 ***** 2021-02-11T06:06:02.661 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-02-11T06:06:02.662 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:02.693 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:02 +0000 (0:00:00.058) 0:02:13.262 ***** 2021-02-11T06:06:02.719 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-02-11T06:06:02.720 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:02.751 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:02 +0000 (0:00:00.057) 0:02:13.319 ***** 2021-02-11T06:06:02.777 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-02-11T06:06:02.778 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:02.811 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:02 +0000 (0:00:00.060) 0:02:13.380 ***** 2021-02-11T06:06:02.838 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-02-11T06:06:02.840 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:02.872 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:02 +0000 (0:00:00.060) 0:02:13.440 ***** 2021-02-11T06:06:02.898 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-02-11T06:06:02.900 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:02.932 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:02 +0000 (0:00:00.059) 0:02:13.500 ***** 2021-02-11T06:06:02.957 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-02-11T06:06:02.959 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:02.993 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:02 +0000 (0:00:00.060) 0:02:13.561 ***** 2021-02-11T06:06:03.020 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:03.062 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:03 +0000 (0:00:00.069) 0:02:13.631 ***** 2021-02-11T06:06:03.089 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-02-11T06:06:03.091 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:03.127 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:03 +0000 (0:00:00.064) 0:02:13.695 ***** 2021-02-11T06:06:03.154 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-02-11T06:06:03.155 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:03.187 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:03 +0000 (0:00:00.060) 0:02:13.756 ***** 2021-02-11T06:06:03.214 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-02-11T06:06:03.215 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:03.249 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:03 +0000 (0:00:00.061) 0:02:13.817 ***** 2021-02-11T06:06:03.277 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2021-02-11T06:06:03.278 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:03.312 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:03 +0000 (0:00:00.063) 0:02:13.881 ***** 2021-02-11T06:06:12.522 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "path": "/etc/ceph", "state": "absent"} Thursday 11 February 2021 06:06:03 +0000 (0:00:00.166) 0:02:14.047 ***** TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "size": 578, "state": "file", "uid": 0} Thursday 11 February 2021 06:06:03 +0000 (0:00:00.341) 0:02:14.389 ***** TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:06:03 +0000 (0:00:00.059) 0:02:14.448 ***** TASK [testnode : Upload weekly kernel-clean crontab.] ************************** ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "checksum": "d5677bbd13857966eb0fc86f66195b18a2b98e20", "dest": "/etc/cron.weekly/kernel-clean", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cron.weekly/kernel-clean", "size": 752, "state": "file", "uid": 0} Thursday 11 February 2021 06:06:04 +0000 (0:00:00.349) 0:02:14.798 ***** TASK [testnode : Install packages] ********************************************* changed: [smithi061.front.sepia.ceph.com] => {"cache_update_time": 1613023167, "cache_updated": false, "changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\ngit is already the newest version (1:2.17.1-1ubuntu0 2021-02-11T06:06:12.524 INFO:teuthology.task.ansible.out:.7).\nThe following additional packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils\nSuggested packages:\n samba vde2 sgabios ovmf debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 6 newly installed, 0 to remove and 19 not upgraded.\nNeed to get 10.4 MB of archives.\nAfter this operation, 47.6 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.6 [2724 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.6 [925 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.35 [40.5 kB]\nGet:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.35 [672 kB]\nGet:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.35 [5194 kB]\nGet:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.35 [868 kB]\nFetched 10.4 MB in 1s (9180 kB/s)\nSelecting previously unselected package librados2.\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137468 files and directories currently installed.)\r\nPreparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.6_amd 2021-02-11T06:06:12.525 INFO:teuthology.task.ansible.out:64.deb ...\r\nUnpacking librados2 (12.2.13-0ubuntu0.18.04.6) ...\r\nSelecting previously unselected package librbd1.\r\nPreparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.6_amd64.deb ...\r\nUnpacking librbd1 (12.2.13-0ubuntu0.18.04.6) ...\r\nSelecting previously unselected package qemu-block-extra:amd64.\r\nPreparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.35_amd64.deb ...\r\nUnpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.35) ...\r\nSelecting previously unselected package qemu-system-common.\r\nPreparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.35_amd64.deb ...\r\nUnpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.35) ...\r\nSelecting previously unselected package qemu-system-x86.\r\nPreparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.35_amd64.deb ...\r\nUnpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.35) ...\r\nSelecting previously unselected package qemu-utils.\r\nPreparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.35_amd64.deb ...\r\nUnpacking qemu-utils (1:2.11+dfsg-1ubuntu7.35) ...\r\nSetting up librados2 (12.2.13-0ubuntu0.18.04.6) ...\r\nSetting up librbd1 (12.2.13-0ubuntu0.18.04.6) ...\r\nSetting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.35) ...\r\nSetting up qemu-utils (1:2.11+dfsg-1ubuntu7.35) ...\r\nSetting up qemu-system-common (1:2.11+dfsg-1ubuntu7.35) ...\r\nSetting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.35) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1.4) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "git is already the newest version (1:2.17.1-1ubuntu0.7).", "The following additional packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils", "Suggested packages:", " samba vde2 sgabios ovmf debootstrap", "The following NEW packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system 2021-02-11T06:06:12.527 INFO:teuthology.task.ansible.out:-common qemu-system-x86", " qemu-utils", "0 upgraded, 6 newly installed, 0 to remove and 19 not upgraded.", "Need to get 10.4 MB of archives.", "After this operation, 47.6 MB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.6 [2724 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.6 [925 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.35 [40.5 kB]", "Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.35 [672 kB]", "Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.35 [5194 kB]", "Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.35 [868 kB]", "Fetched 10.4 MB in 1s (9180 kB/s)", "Selecting previously unselected package librados2.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137468 files and directories currently installed.)", "Preparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.6_amd64.deb ...", "Unpacking librados2 (12.2.13-0ubuntu0.18.04.6) ...", "Selecting previously unselected package librbd1.", "Preparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.6_amd64.deb ...", "Unpacking librbd1 (12 2021-02-11T06:06:12.528 INFO:teuthology.task.ansible.out:.2.13-0ubuntu0.18.04.6) ...", "Selecting previously unselected package qemu-block-extra:amd64.", "Preparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.35_amd64.deb ...", "Unpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.35) ...", "Selecting previously unselected package qemu-system-common.", "Preparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.35_amd64.deb ...", "Unpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.35) ...", "Selecting previously unselected package qemu-system-x86.", "Preparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.35_amd64.deb ...", "Unpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.35) ...", "Selecting previously unselected package qemu-utils.", "Preparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.35_amd64.deb ...", "Unpacking qemu-utils (1:2.11+dfsg-1ubuntu7.35) ...", "Setting up librados2 (12.2.13-0ubuntu0.18.04.6) ...", "Setting up librbd1 (12.2.13-0ubuntu0.18.04.6) ...", "Setting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.35) ...", "Setting up qemu-utils (1:2.11+dfsg-1ubuntu7.35) ...", "Setting up qemu-system-common (1:2.11+dfsg-1ubuntu7.35) ...", "Setting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.35) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.4) ..."]} Thursday 11 February 2021 06:06:04 +0000 (0:00:00.336) 0:02:15.134 ***** TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "checksum": "22e9a1446f43d0440219b9af14a66dd77fb0c2eb", "dest": "/etc/grub.d/02_force_timeout", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/grub.d/02_force_timeout", "size": 115, "state": "file", "uid": 0} Thursday 11 February 2021 06:06:04 +0000 (0:00:00.044) 0:02:15.178 ***** TASK [testnode : Enable kernel modules to load at boot time.] ****************** ok: [smithi172.front.sepia.ceph.com] => {"chang 2021-02-11T06:06:12.529 INFO:teuthology.task.ansible.out:ed": false, "checksum": "6449466d762bced58744df80432d189e35329803", "dest": "/etc/modules", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/modules", "size": 315, "state": "file", "uid": 0} Thursday 11 February 2021 06:06:04 +0000 (0:00:00.330) 0:02:15.509 ***** TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** ok: [smithi172.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} Thursday 11 February 2021 06:06:05 +0000 (0:00:00.163) 0:02:15.673 ***** TASK [testnode : Work around broken wget on wheezy.] *************************** skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:06:05 +0000 (0:00:00.061) 0:02:15.735 ***** TASK [testnode : Stop collectl] ************************************************ skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:06:05 +0000 (0:00:00.064) 0:02:15.799 ***** TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:06:05 +0000 (0:00:00.059) 0:02:15.858 ***** TASK [testnode : Check to see if we need to edit /etc/profile.] **************** skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:06:05 +0000 (0:00:00.060) 0:02:15.919 ***** TASK [testnode : Update /etc/profile if needed.] ******************************* skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:06:05 +0000 (0:00:00.066) 0:02:15.986 ***** TASK [testnode : set_fact] ************************** 2021-02-11T06:06:12.530 INFO:teuthology.task.ansible.out:*************************** ok: [smithi172.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} Thursday 11 February 2021 06:06:05 +0000 (0:00:00.064) 0:02:16.050 ***** TASK [testnode : Install non aarch64 packages] ********************************* ok: [smithi061.front.sepia.ceph.com] => {"cache_update_time": 1613023167, "cache_updated": false, "changed": false} Thursday 11 February 2021 06:06:05 +0000 (0:00:00.044) 0:02:16.095 ***** TASK [testnode : Check for NVMe drive] ***************************************** skipping: [smithi172.front.sepia.ceph.com] => (item=loop1) => {"ansible_loop_var": "item", "changed": false, "item": "loop1", "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item=loop6) => {"ansible_loop_var": "item", "changed": false, "item": "loop6", "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item=loop4) => {"ansible_loop_var": "item", "changed": false, "item": "loop4", "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} ok: [smithi172.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} skipping: [smithi172.front.sepia.ceph.com] => (item=loop3) => {"ansible_loop_var": "item", "changed": false, "item": "loop3", "skip_reason": "Conditional result was False"} skipping: [smithi172.front. 2021-02-11T06:06:12.532 INFO:teuthology.task.ansible.out:sepia.ceph.com] => (item=loop2) => {"ansible_loop_var": "item", "changed": false, "item": "loop2", "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item=loop0) => {"ansible_loop_var": "item", "changed": false, "item": "loop0", "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item=loop7) => {"ansible_loop_var": "item", "changed": false, "item": "loop7", "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item=loop5) => {"ansible_loop_var": "item", "changed": false, "item": "loop5", "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-02-11T06:06:12.532 INFO:teuthology.task.ansible.out: 2021-02-11T06:06:12.534 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2021-02-11T06:06:12.536 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "cmd": ["perl", "-pe", "if (m{^([^#]\\S*\\s+/\\s+\\S+\\s+)(\\S+)(\\s+.*)$}) { $_=\"$1$2,user_xattr$3\\n\" unless $2=~m{(^|,)user_xattr(,|$)}; }", "-i.bak", "/etc/fstab"], "rc": 0, "stdout": "skipped, since /etc/fstab.bak exists", "stdout_lines": ["skipped, since /etc/fstab.bak exists"]} 2021-02-11T06:06:12.569 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:12 +0000 (0:00:07.043) 0:02:23.138 ***** 2021-02-11T06:06:12.605 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:12 +0000 (0:00:00.035) 0:02:23.173 ***** 2021-02-11T06:06:12.610 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2021-02-11T06:06:12.610 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:12.647 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:12 +0000 (0:00:00.041) 0:02:23.215 ***** 2021-02-11T06:06:12.776 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2021-02-11T06:06:12.777 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "fuse", "state": "present", "system": false} 2021-02-11T06:06:12.812 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:12 +0000 (0:00:00.165) 0:02:23.381 ***** 2021-02-11T06:06:12.829 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2021-02-11T06:06:12.830 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.008553", "end": "2021-02-11 06:06:12.796955", "rc": 0, "start": "2021-02-11 06:06:12.788402", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/bin/teuthology-lock"]} 2021-02-11T06:06:12.867 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:12 +0000 (0:00:00.054) 0:02:23.436 ***** 2021-02-11T06:06:12.893 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2021-02-11T06:06:12.894 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:12.929 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:12 +0000 (0:00:00.062) 0:02:23.498 ***** 2021-02-11T06:06:12.967 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2021-02-11T06:06:12.968 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:13.006 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:13 +0000 (0:00:00.076) 0:02:23.574 ***** 2021-02-11T06:06:13.032 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2021-02-11T06:06:13.033 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:13.065 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:13 +0000 (0:00:00.058) 0:02:23.633 ***** 2021-02-11T06:06:13.122 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2021-02-11T06:06:13.123 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "checksum": "9edff47031d0f14a9b8c813b44eb4a9296d0439f", "dest": "/etc/fuse.conf", "gid": 10102, "group": "fuse", "mode": "0644", "owner": "root", "path": "/etc/fuse.conf", "size": 366, "state": "file", "uid": 0} 2021-02-11T06:06:13.158 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:13 +0000 (0:00:00.093) 0:02:23.726 ***** 2021-02-11T06:06:13.328 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2021-02-11T06:06:14.034 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1238, "groups": "fuse,kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} Thursday 11 February 2021 06:06:13 +0000 (0:00:00.206) 0:02:23.933 ***** TASK [testnode : Set up static IP in /etc/hosts.] ****************************** ok: [smithi061.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} Thursday 11 February 2021 06:06:13 +0000 (0:00:00.165) 0:02:24.098 ***** TASK [testnode : Stop apache2] ************************************************* changed: [smithi061.front.sepia.ceph.com] => {"changed": true, "name": "apache2", "state": "stopped", "status": {"ActiveEnterTimestamp": "Thu 2021-02-11 06:03:11 UTC", "ActiveEnterTimestampMonotonic": "20380587", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "remote-fs.target -.mount system.slice systemd-tmpfiles-setup.service basic.target sysinit.target network.target nss-lookup.target systemd-journald.socket", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2021-02-11 06:03:10 UTC", "AssertTimestampMonotonic": "19945368", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_ch 2021-02-11T06:06:14.036 INFO:teuthology.task.ansible.out:root cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2021-02-11 06:03:10 UTC", "ConditionTimestampMonotonic": "19945368", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/apache2.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "The Apache HTTP Server", "DevicePolicy": "auto", "DropInPaths": "/lib/systemd/system/apache2.service.d/apache2-systemd.conf", "DynamicUser": "no", "Environment": "APACHE_STARTED_BY_SYSTEMD=true", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1350", "ExecMainStartTimestamp": "Thu 2021-02-11 06:03:11 UTC", "ExecMainStartTimestampMonotonic": "20380568", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/apache2.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "apache2 2021-02-11T06:06:14.037 INFO:teuthology.task.ansible.out:.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2021-02-11 06:03:10 UTC", "InactiveExitTimestampMonotonic": "19946691", "InvocationID": "c024235de7b7434193669787f2630500", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128029", "LimitNPROCSoft": "128029", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "128029", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1350", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "apache2.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "Priv 2021-02-11T06:06:14.038 INFO:teuthology.task.ansible.out:ateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount sysinit.target system.slice", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2021-02-11 06:03:11 UTC", "StateChangeTimestampMonotonic": "20380587", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "55", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2021-02-11 06:03:11 UTC", "WatchdogTimestampMonotonic": "20380585", "WatchdogUSec": "0"}} Thursday 11 February 2021 06:06:13 +0000 (0:00:00.337) 0:02:24.436 2021-02-11T06:06:14.040 INFO:teuthology.task.ansible.out:***** TASK [testnode : Set mode on /etc/fuse.conf] *********************************** skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:06:13 +0000 (0:00:00.058) 0:02:24.495 ***** TASK [testnode : Make sure apt dependencies are installed] ********************* ok: [smithi172.front.sepia.ceph.com] => {"cache_update_time": 1613023222, "cache_updated": false, "changed": false} Thursday 11 February 2021 06:06:13 +0000 (0:00:00.047) 0:02:24.542 ***** TASK [testnode : Ensure the group kvm exists.] ********************************* skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:06:14 +0000 (0:00:00.039) 0:02:24.582 ***** TASK [testnode : Make sure rpm dependencies are installed] ********************* skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:14.054 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:14 +0000 (0:00:00.040) 0:02:24.622 ***** 2021-02-11T06:06:14.059 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-02-11T06:06:14.061 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:14.096 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:14 +0000 (0:00:00.042) 0:02:24.664 ***** 2021-02-11T06:06:14.102 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-02-11T06:06:14.103 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item={'block_used': 13754882, 'uuid': '602a9f0b-ccf6-4d5d-877a-542e0952c07a', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226340226, 'size_available': 927089565696, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149545, 'block_size': 4096, 'inode_available': 60905431}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226340226, "block_size": 4096, "block_total": 240095108, "block_used": 13754882, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905431, "inode_total": 61054976, "inode_used": 149545, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927089565696, "size_total": 983429562368, "uuid": "602a9f0b-ccf6-4d5d-877a-542e0952c07a"}} 2021-02-11T06:06:14.137 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:14 +0000 (0:00:00.041) 0:02:24.705 ***** 2021-02-11T06:06:14.143 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-02-11T06:06:14.143 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:14.179 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:14 +0000 (0:00:00.042) 0:02:24.747 ***** 2021-02-11T06:06:14.207 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-02-11T06:06:14.208 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:14.240 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:14 +0000 (0:00:00.061) 0:02:24.809 ***** 2021-02-11T06:06:14.266 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-02-11T06:06:14.268 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Compile list of non-root partitions] ************************** changed: [smithi172.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.008344", "end": "2021-02-11 06:06:14.246389", "rc": 0, "start": "2021-02-11 06:06:14.238045", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-11T06:06:14.301 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:14 +0000 (0:00:00.060) 0:02:24.869 ***** 2021-02-11T06:06:14.332 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:14 +0000 (0:00:00.031) 0:02:24.900 ***** 2021-02-11T06:06:14.338 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-02-11T06:06:14.340 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:14.373 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:14 +0000 (0:00:00.040) 0:02:24.941 ***** 2021-02-11T06:06:14.378 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2021-02-11T06:06:14.379 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'block_used': 13754882, 'uuid': '602a9f0b-ccf6-4d5d-877a-542e0952c07a', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226340226, 'size_available': 927089565696, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149545, 'block_size': 4096, 'inode_available': 60905431}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226340226, "block_size": 4096, "block_total": 240095108, "block_used": 13754882, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905431, "inode_total": 61054976, "inode_used": 149545, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927089565696, "size_total": 983429562368, "uuid": "602a9f0b-ccf6-4d5d-877a-542e0952c07a"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:14.417 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:14 +0000 (0:00:00.044) 0:02:24.986 ***** 2021-02-11T06:06:14.422 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-02-11T06:06:14.424 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:14.456 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:14 +0000 (0:00:00.039) 0:02:25.025 ***** 2021-02-11T06:06:14.463 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2021-02-11T06:06:14.497 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:14 +0000 (0:00:00.040) 0:02:25.066 ***** 2021-02-11T06:06:14.503 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-11T06:06:14.504 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:14.542 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:14 +0000 (0:00:00.044) 0:02:25.110 ***** 2021-02-11T06:06:14.549 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-02-11T06:06:14.550 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:14.561 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:14.595 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:14 +0000 (0:00:00.053) 0:02:25.164 ***** 2021-02-11T06:06:14.621 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-02-11T06:06:14.622 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:14.654 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:14 +0000 (0:00:00.058) 0:02:25.223 ***** 2021-02-11T06:06:14.681 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:14.716 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:14 +0000 (0:00:00.061) 0:02:25.284 ***** 2021-02-11T06:06:14.743 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:14.776 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:14 +0000 (0:00:00.060) 0:02:25.344 ***** 2021-02-11T06:06:14.801 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-02-11T06:06:14.803 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:14.834 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:14 +0000 (0:00:00.058) 0:02:25.403 ***** 2021-02-11T06:06:14.859 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-02-11T06:06:14.861 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:14.893 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:14 +0000 (0:00:00.058) 0:02:25.461 ***** 2021-02-11T06:06:14.919 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-02-11T06:06:14.920 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:14.954 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:14 +0000 (0:00:00.061) 0:02:25.523 ***** 2021-02-11T06:06:14.982 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-02-11T06:06:15.513 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:06:15 +0000 (0:00:00.063) 0:02:25.586 ***** TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:06:15 +0000 (0:00:00.062) 0:02:25.648 ***** TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:06:15 +0000 (0:00:00.065) 0:02:25.714 ***** TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:06:15 +0000 (0:00:00.063) 0:02:25.777 ***** TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:06:15 +0000 (0:00:00.062) 0:02:25.839 ***** TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:06:15 +0000 (0:00:00.060) 0:02:25.900 ***** TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:06:15 +0000 (0:00:00.060) 0:02:25.961 ***** TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [smithi061.front.sepia.ceph.com] => { 2021-02-11T06:06:15.514 INFO:teuthology.task.ansible.out:"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:06:15 +0000 (0:00:00.058) 0:02:26.019 ***** TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:06:15 +0000 (0:00:00.060) 0:02:26.079 ***** 2021-02-11T06:06:15.538 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2021-02-11T06:06:15.539 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:15.570 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:15 +0000 (0:00:00.059) 0:02:26.139 ***** 2021-02-11T06:06:15.700 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2021-02-11T06:06:15.701 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "path": "/etc/ceph", "state": "absent"} 2021-02-11T06:06:15.736 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:15 +0000 (0:00:00.165) 0:02:26.304 ***** 2021-02-11T06:06:16.043 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-02-11T06:06:16.045 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "size": 578, "state": "file", "uid": 0} 2021-02-11T06:06:16.077 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:16 +0000 (0:00:00.340) 0:02:26.645 ***** 2021-02-11T06:06:16.102 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-02-11T06:06:16.104 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:16.136 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:16 +0000 (0:00:00.059) 0:02:26.705 ***** 2021-02-11T06:06:16.195 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-02-11T06:06:16.196 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-jToQa2fQrO7ZROVQCq7wn3R2X9OzSMrlPe7qMq7gvrIIGzHsnHtCQnVjmTqPyxei'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.534700", "end": "2021-02-11 06:06:16.164006", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-jToQa2fQrO7ZROVQCq7wn3R2X9OzSMrlPe7qMq7gvrIIGzHsnHtCQnVjmTqPyxei"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 06:06:14.629306", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-11T06:06:16.208 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:16.219 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:16.637 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2021-02-11T06:06:32.572 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "checksum": "d5677bbd13857966eb0fc86f66195b18a2b98e20", "dest": "/etc/cron.weekly/kernel-clean", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cron.weekly/kernel-clean", "size": 752, "state": "file", "uid": 0} Thursday 11 February 2021 06:06:16 +0000 (0:00:00.533) 0:02:27.238 ***** TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "checksum": "22e9a1446f43d0440219b9af14a66dd77fb0c2eb", "dest": "/etc/grub.d/02_force_timeout", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/grub.d/02_force_timeout", "size": 115, "state": "file", "uid": 0} Thursday 11 February 2021 06:06:17 +0000 (0:00:00.340) 0:02:27.578 ***** TASK [testnode : Enable kernel modules to load at boot time.] ****************** ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "checksum": "6449466d762bced58744df80432d189e35329803", "dest": "/etc/modules", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/modules", "size": 315, "state": "file", "uid": 0} Thursday 11 February 2021 06:06:17 +0000 (0:00:00.335) 0:02:27.914 ***** TASK [testnode : Zap all non-root disks] *************************************** changed: [smithi172.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-jToQa2fQrO7ZROVQCq7wn3R2X9OzSMrlZWudP29Vtq6kYM8HitVdQEPw5amKb9NC'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm 2021-02-11T06:06:32.574 INFO:teuthology.task.ansible.out:-0", "delta": "0:00:01.138506", "end": "2021-02-11 06:06:17.437976", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-jToQa2fQrO7ZROVQCq7wn3R2X9OzSMrlZWudP29Vtq6kYM8HitVdQEPw5amKb9NC"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 06:06:16.299470", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** ok: [smithi061.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} TASK [testnode : Zap all non-root disks] *************************************** skipping: [smithi172.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XNKA', 'wwn-0x5000c5009294065e'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '602a9f0b-ccf6-4d5d-877a-542e0952c07a', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XNKA-part1', 'wwn-0x5000c5009294065e-part1'], 'uuids': ['60 2021-02-11T06:06:32.575 INFO:teuthology.task.ansible.out:2a9f0b-ccf6-4d5d-877a-542e0952c07a']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c5009294065e', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5XNKA', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XNKA", "wwn-0x5000c5009294065e"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XNKA-part1", "wwn-0x5000c5009294065e-part1"], "labels": [], "masters": [], "uuids": ["602a9f0b-ccf6-4d5d-877a-542e0952c07a"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "602a9f0b-ccf6-4d5d-877a-542e0952c07a"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XNKA", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c5009294065e"}}, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:06:17 +0000 (0:00:00.162) 0:02:28.076 ***** TASK [testnode : Work around broken wget on wheezy.] *************************** skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:06:17 +0000 (0:00:00.062) 0:02:28.138 ***** TASK [testnode : Stop collectl] ************************************************ skipping: [smithi061.front.sepia.ce 2021-02-11T06:06:32.576 INFO:teuthology.task.ansible.out:ph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:06:17 +0000 (0:00:00.062) 0:02:28.201 ***** TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:06:17 +0000 (0:00:00.071) 0:02:28.273 ***** TASK [testnode : Check to see if we need to edit /etc/profile.] **************** skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:06:17 +0000 (0:00:00.061) 0:02:28.335 ***** TASK [testnode : Update /etc/profile if needed.] ******************************* skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:06:17 +0000 (0:00:00.060) 0:02:28.395 ***** TASK [testnode : set_fact] ***************************************************** ok: [smithi061.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} Thursday 11 February 2021 06:06:17 +0000 (0:00:00.073) 0:02:28.469 ***** TASK [testnode : Check for NVMe drive] ***************************************** skipping: [smithi061.front.sepia.ceph.com] => (item=loop1) => {"ansible_loop_var": "item", "changed": false, "item": "loop1", "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item=loop6) => {"ansible_loop_var": "item", "changed": false, "item": "loop6", "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item=loop4) => {"ansible_loop_var": "item", "changed": false, "item": "loop 2021-02-11T06:06:32.578 INFO:teuthology.task.ansible.out:4", "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} ok: [smithi061.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} skipping: [smithi061.front.sepia.ceph.com] => (item=loop3) => {"ansible_loop_var": "item", "changed": false, "item": "loop3", "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item=loop2) => {"ansible_loop_var": "item", "changed": false, "item": "loop2", "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item=loop0) => {"ansible_loop_var": "item", "changed": false, "item": "loop0", "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item=loop7) => {"ansible_loop_var": "item", "changed": false, "item": "loop7", "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item=loop5) => {"ansible_loop_var": "item", "changed": false, "item": "loop5", "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:06:18 +00 2021-02-11T06:06:32.579 INFO:teuthology.task.ansible.out:00 (0:00:00.208) 0:02:28.677 ***** TASK [testnode : Check for teuthology-lock command] **************************** changed: [smithi061.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003073", "end": "2021-02-11 06:06:18.276919", "rc": 0, "start": "2021-02-11 06:06:18.273846", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/bin/teuthology-lock"]} Thursday 11 February 2021 06:06:18 +0000 (0:00:00.243) 0:02:28.921 ***** TASK [testnode : Mark system down if NVMe card missing] ************************ skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:06:18 +0000 (0:00:00.067) 0:02:28.988 ***** TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:06:18 +0000 (0:00:00.063) 0:02:29.052 ***** TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:06:18 +0000 (0:00:00.063) 0:02:29.115 ***** TASK [testnode : Zap all non-root disks] *************************************** changed: [smithi172.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-Pzsdtw-KGr4-Yw1h-i8BM-h4gE-ND1F-mhbrUN', 'nvme-INTEL_SSDPEDMD400G4_CVFT623 2021-02-11T06:06:32.580 INFO:teuthology.task.ansible.out:3005M400BGN', 'nvme-nvme.8086-43564654363233333030354d34303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.165976", "end": "2021-02-11 06:06:18.728606", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-Pzsdtw-KGr4-Yw1h-i8BM-h4gE-ND1F-mhbrUN", "nvme-INTEL_SSDPEDMD400G4_CVFT6233005M400BGN", "nvme-nvme.8086-43564654363233333030354d34303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 06:06:17.562630", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} skipping: [smithi172.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 2021-02-11T06:06:32.581 INFO:teuthology.task.ansible.out: 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} TASK [testnode : Make sure apt dependencies are installed] ********************* ok: [smithi061.front.sepia.ceph.com] => {"cache_update_time": 1613023167, "cache_updated": false, "changed": false} Thursday 11 February 2021 06:06:19 +0000 (0:00:01.022) 0:02:30.137 ***** TASK [testno 2021-02-11T06:06:32.583 INFO:teuthology.task.ansible.out:de : Make sure rpm dependencies are installed] ********************* skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:06:19 +0000 (0:00:00.064) 0:02:30.202 ***** TASK [testnode : Set root disk] ************************************************ ok: [smithi061.front.sepia.ceph.com] => (item={'block_used': 13754885, 'uuid': '602a9f0b-ccf6-4d5d-877a-542e0952c07a', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226340223, 'size_available': 927089553408, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149546, 'block_size': 4096, 'inode_available': 60905430}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": 2021-02-11T06:06:32.583 INFO:teuthology.task.ansible.out:"item", "changed": false, "item": {"block_available": 226340223, "block_size": 4096, "block_total": 240095108, "block_used": 13754885, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905430, "inode_total": 61054976, "inode_used": 149546, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927089553408, "size_total": 983429562368, "uuid": "602a9f0b-ccf6-4d5d-877a-542e0952c07a"}} 2021-02-11T06:06:32.586 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-02-11T06:06:32.586 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-jToQa2fQrO7ZROVQCq7wn3R2X9OzSMrlQ7vtPRmkKqhncxWTBjB1a6MNhELu0PhI'], 'uuids': ['9e756502-992a-4e2e-9847-e645484ad7a1']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.154918", "end": "2021-02-11 06:06:20.021232", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-jToQa2fQrO7ZROVQCq7wn3R2X9OzSMrlQ7vtPRmkKqhncxWTBjB1a6MNhELu0PhI"], "labels": [], "masters": [], "uuids": ["9e756502-992a-4e2e-9847-e645484ad7a1"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 06:06:18.866314", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-11T06:06:32.587 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:32.589 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:32.590 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-jToQa2fQrO7ZROVQCq7wn3R2X9OzSMrl9x6OjCPrxPGUDRmyl42xUjOjuLiBA5B0'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.129080", "end": "2021-02-11 06:06:21.312523", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-jToQa2fQrO7ZROVQCq7wn3R2X9OzSMrl9x6OjCPrxPGUDRmyl42xUjOjuLiBA5B0"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 06:06:20.183443", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-11T06:06:32.591 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:32.593 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-jToQa2fQrO7ZROVQCq7wn3R2X9OzSMrlWvpRsF1EynHECm29VkTr96ebDCsPIogY'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.128821", "end": "2021-02-11 06:06:22.571946", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-jToQa2fQrO7ZROVQCq7wn3R2X9OzSMrlWvpRsF1EynHECm29VkTr96ebDCsPIogY"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 06:06:21.443125", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-11T06:06:32.688 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:32 +0000 (0:00:13.053) 0:02:43.256 ***** 2021-02-11T06:06:32.726 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:32 +0000 (0:00:00.037) 0:02:43.294 ***** 2021-02-11T06:06:32.787 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-02-11T06:06:32.788 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:32.881 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2021-02-11T06:06:32.883 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.009960", "end": "2021-02-11 06:06:32.682240", "rc": 0, "start": "2021-02-11 06:06:32.672280", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-11T06:06:32.919 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:32 +0000 (0:00:00.193) 0:02:43.488 ***** 2021-02-11T06:06:32.973 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2021-02-11T06:06:32.975 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'block_used': 13754885, 'uuid': '602a9f0b-ccf6-4d5d-877a-542e0952c07a', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226340223, 'size_available': 927089553408, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149546, 'block_size': 4096, 'inode_available': 60905430}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226340223, "block_size": 4096, "block_total": 240095108, "block_used": 13754885, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905430, "inode_total": 61054976, "inode_used": 149546, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927089553408, "size_total": 983429562368, "uuid": "602a9f0b-ccf6-4d5d-877a-542e0952c07a"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:33.011 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:33 +0000 (0:00:00.091) 0:02:43.579 ***** 2021-02-11T06:06:33.038 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2021-02-11T06:06:33.076 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:33 +0000 (0:00:00.064) 0:02:43.644 ***** 2021-02-11T06:06:33.084 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-02-11T06:06:33.085 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-jToQa2fQrO7ZROVQCq7wn3R2X9OzSMrlPe7qMq7gvrIIGzHsnHtCQnVjmTqPyxei'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.156856", "end": "2021-02-11 06:06:33.037061", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-jToQa2fQrO7ZROVQCq7wn3R2X9OzSMrlPe7qMq7gvrIIGzHsnHtCQnVjmTqPyxei"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 06:06:32.880205", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.152056 s, 759 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.152056 s, 759 MB/s"], "stdout": "", "stdout_lines": []} 2021-02-11T06:06:33.097 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:33.098 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:33.120 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-02-11T06:06:33.121 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:33.373 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-02-11T06:06:33.373 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-jToQa2fQrO7ZROVQCq7wn3R2X9OzSMrlZWudP29Vtq6kYM8HitVdQEPw5amKb9NC'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.156677", "end": "2021-02-11 06:06:33.340472", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-jToQa2fQrO7ZROVQCq7wn3R2X9OzSMrlZWudP29Vtq6kYM8HitVdQEPw5amKb9NC"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 06:06:33.183795", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.152011 s, 759 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.152011 s, 759 MB/s"], "stdout": "", "stdout_lines": []} 2021-02-11T06:06:33.385 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XNKA', 'wwn-0x5000c5009294065e'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '602a9f0b-ccf6-4d5d-877a-542e0952c07a', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XNKA-part1', 'wwn-0x5000c5009294065e-part1'], 'uuids': ['602a9f0b-ccf6-4d5d-877a-542e0952c07a']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c5009294065e', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5XNKA', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XNKA", "wwn-0x5000c5009294065e"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XNKA-part1", "wwn-0x5000c5009294065e-part1"], "labels": [], "masters": [], "uuids": ["602a9f0b-ccf6-4d5d-877a-542e0952c07a"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "602a9f0b-ccf6-4d5d-877a-542e0952c07a"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XNKA", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c5009294065e"}}, "skip_reason": 2021-02-11T06:06:33.387 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2021-02-11T06:06:33.565 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-Pzsdtw-KGr4-Yw1h-i8BM-h4gE-ND1F-mhbrUN', 'nvme-INTEL_SSDPEDMD400G4_CVFT6233005M400BGN', 'nvme-nvme.8086-43564654363233333030354d34303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.054162", "end": "2021-02-11 06:06:33.529475", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-Pzsdtw-KGr4-Yw1h-i8BM-h4gE-ND1F-mhbrUN", "nvme-INTEL_SSDPEDMD400G4_CVFT6233005M400BGN", "nvme-nvme.8086-43564654363233333030354d34303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 06:06:33.475313", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0493307 s, 2.3 GB/s", "std 2021-02-11T06:06:57.891 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0493307 s, 2.3 GB/s"], "stdout": "", "stdout_lines": []} skipping: [smithi172.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} changed: [smithi1 2021-02-11T06:06:57.893 INFO:teuthology.task.ansible.out:72.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-jToQa2fQrO7ZROVQCq7wn3R2X9OzSMrlQ7vtPRmkKqhncxWTBjB1a6MNhELu0PhI'], 'uuids': ['9e756502-992a-4e2e-9847-e645484ad7a1']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.159374", "end": "2021-02-11 06:06:33.831441", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-jToQa2fQrO7ZROVQCq7wn3R2X9OzSMrlQ7vtPRmkKqhncxWTBjB1a6MNhELu0PhI"], "labels": [], "masters": [], "uuids": ["9e756502-992a-4e2e-9847-e645484ad7a1"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 06:06:33.672067", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.15687 s, 735 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.15687 s, 735 MB/s"], "stdout": "", "stdout_lines": []} skipping: [smithi172.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => 2021-02-11T06:06:57.894 INFO:teuthology.task.ansible.out:{"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} changed: [smithi172.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-jToQa2fQrO7ZROVQCq7wn3R2X9OzSMrl9x6OjCPrxPGUDRmyl42xUjOjuLiBA5B0'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero o 2021-02-11T06:06:57.895 INFO:teuthology.task.ansible.out:f=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.172038", "end": "2021-02-11 06:06:34.143553", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-jToQa2fQrO7ZROVQCq7wn3R2X9OzSMrl9x6OjCPrxPGUDRmyl42xUjOjuLiBA5B0"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 06:06:33.971515", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.167326 s, 689 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.167326 s, 689 MB/s"], "stdout": "", "stdout_lines": []} skipping: [smithi172.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} changed: [smithi172.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 2021-02-11T06:06:57.897 INFO:teuthology.task.ansible.out: 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-jToQa2fQrO7ZROVQCq7wn3R2X9OzSMrlWvpRsF1EynHECm29VkTr96ebDCsPIogY'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.156027", "end": "2021-02-11 06:06:34.431116", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-jToQa2fQrO7ZROVQCq7wn3R2X9OzSMrlWvpRsF1EynHECm29VkTr96ebDCsPIogY"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 06:06:34.275089", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.151512 s, 761 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.151512 s, 761 MB/s"], "stdout": "", "stdout_lines": []} Thursday 11 February 2021 06:06:34 +0000 (0:00:01.436) 0:02:45.081 ***** TASK [testnode : Remove all LVM data] ****************************************** changed: [smithi172.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.124721", "end": "2021-02-11 06:06:34.748649", "rc": 0, "start": "2021-02-11 06:06:34.623928", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} Thursday 11 February 2021 06:06:34 +0000 (0:00:00.303) 0:02:45.384 ***** TASK [testnode : Zap all non-root disks] *************************************** changed: [smithi061.front.sepia.ceph.com] 2021-02-11T06:06:57.898 INFO:teuthology.task.ansible.out:=> (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-Y7sQo6iFEup7aLZOCZRt0qATIEk8C2RxnIia0Oo8HPoIjJHOXDrm2VHCx4J3tldq'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.575603", "end": "2021-02-11 06:06:34.799405", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-Y7sQo6iFEup7aLZOCZRt0qATIEk8C2RxnIia0Oo8HPoIjJHOXDrm2VHCx4J3tldq"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 06:06:33.223802", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} skipping: [smithi061.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': 2021-02-11T06:06:57.899 INFO:teuthology.task.ansible.out: {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} TASK [testnode : Check for physical volumes] *********************************** changed: [smithi172.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.065723", "end": "2021-02-11 06:06:34.985420", "rc": 0, "start": "2021-02-11 06:06:34.919697", "stderr": "", "stderr 2021-02-11T06:06:57.901 INFO:teuthology.task.ansible.out:_lines": [], "stdout": "", "stdout_lines": []} Thursday 11 February 2021 06:06:35 +0000 (0:00:00.230) 0:02:45.614 ***** TASK [testnode : Remove physical volumes] ************************************** Thursday 11 February 2021 06:06:35 +0000 (0:00:00.063) 0:02:45.678 ***** TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:06:35 +0000 (0:00:00.080) 0:02:45.758 ***** TASK [testnode : Write partition entries to {{ item.0.device }}] *************** skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:06:35 +0000 (0:00:00.065) 0:02:45.824 ***** TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:06:35 +0000 (0:00:00.061) 0:02:45.885 ***** TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:57.901 INFO:teuthology.task.ansible.out: 2021-02-11T06:06:57.902 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-02-11T06:06:57.903 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-Y7sQo6iFEup7aLZOCZRt0qATIEk8C2Rxx22ulApDMD94HUbQi42OEPVJ6b30kOiP'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.106595", "end": "2021-02-11 06:06:36.048682", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-Y7sQo6iFEup7aLZOCZRt0qATIEk8C2Rxx22ulApDMD94HUbQi42OEPVJ6b30kOiP"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 06:06:34.942087", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-11T06:06:57.904 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4W2W8', 'wwn-0x5000c50087b8cd81'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '602a9f0b-ccf6-4d5d-877a-542e0952c07a', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4W2W8-part1', 'wwn-0x5000c50087b8cd81-part1'], 'uuids': ['602a9f0b-ccf6-4d5d-877a-542e0952c07a']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c50087b8cd81', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W4W2W8', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4W2W8", "wwn-0x5000c50087b8cd81"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4W2W8-part1", "wwn-0x5000c50087b8cd81-part1"], "labels": [], "masters": [], "uuids": ["602a9f0b-ccf6-4d5d-877a-542e0952c07a"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "602a9f0b-ccf6-4d5d-877a-542e0952c07a"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4W2W8", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50087b8cd81"}}, "skip_reason": 2021-02-11T06:06:57.905 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2021-02-11T06:06:57.905 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-HvUORG-0471-0h9g-pg8G-IPNY-n25z-2M7NWP', 'nvme-INTEL_SSDPEDMD400G4_CVFT534300NV400BGN', 'nvme-nvme.8086-435646543533343330304e5634303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.150425", "end": "2021-02-11 06:06:37.330880", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-HvUORG-0471-0h9g-pg8G-IPNY-n25z-2M7NWP", "nvme-INTEL_SSDPEDMD400G4_CVFT534300NV400BGN", "nvme-nvme.8086-435646543533343330304e5634303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 06:06:36.180455", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries. 2021-02-11T06:06:57.907 INFO:teuthology.task.ansible.out:\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} skipping: [smithi061.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:57.908 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:57.909 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-Y7sQo6iFEup7aLZOCZRt0qATIEk8C2RxJZg5vwmlNBFYyKUiRfge4l10G3ZDZytH'], 'uuids': ['d5154754-1be6-4140-89ee-f9489115f8f5']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.140944", "end": "2021-02-11 06:06:38.608311", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-Y7sQo6iFEup7aLZOCZRt0qATIEk8C2RxJZg5vwmlNBFYyKUiRfge4l10G3ZDZytH"], "labels": [], "masters": [], "uuids": ["d5154754-1be6-4140-89ee-f9489115f8f5"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 06:06:37.467367", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-11T06:06:57.911 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:57.911 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:57.912 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-Y7sQo6iFEup7aLZOCZRt0qATIEk8C2RxQaLeB7vDpb5bhEnYavqBqT3KNwC2qbQ7'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.127797", "end": "2021-02-11 06:06:39.875149", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-Y7sQo6iFEup7aLZOCZRt0qATIEk8C2RxQaLeB7vDpb5bhEnYavqBqT3KNwC2qbQ7"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 06:06:38.747352", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-11T06:06:57.913 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:57.915 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-Y7sQo6iFEup7aLZOCZRt0qATIEk8C2RxBJxzfXsmi0vplctt4m52u2HtFIIPM9PI'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.110454", "end": "2021-02-11 06:06:41.117636", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-Y7sQo6iFEup7aLZOCZRt0qATIEk8C2RxBJxzfXsmi0vplctt4m52u2HtFIIPM9PI"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 06:06:40.007182", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-11T06:06:57.961 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:57 +0000 (0:00:22.644) 0:03:08.529 ***** 2021-02-11T06:06:57.996 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:57 +0000 (0:00:00.035) 0:03:08.565 ***** 2021-02-11T06:06:58.014 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-02-11T06:06:58.019 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:58.154 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2021-02-11T06:06:58.156 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-11T06:06:58.190 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:58 +0000 (0:00:00.194) 0:03:08.759 ***** 2021-02-11T06:06:58.250 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-02-11T06:06:58.251 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'block_used': 13754882, 'uuid': '602a9f0b-ccf6-4d5d-877a-542e0952c07a', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226340226, 'size_available': 927089565696, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149545, 'block_size': 4096, 'inode_available': 60905431}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226340226, "block_size": 4096, "block_total": 240095108, "block_used": 13754882, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905431, "inode_total": 61054976, "inode_used": 149545, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927089565696, "size_total": 983429562368, "uuid": "602a9f0b-ccf6-4d5d-877a-542e0952c07a"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:58.289 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:58 +0000 (0:00:00.098) 0:03:08.857 ***** 2021-02-11T06:06:58.326 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2021-02-11T06:06:58.327 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** changed: [smithi061.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-Y7sQo6iFEup7aLZOCZRt0qATIEk8C2RxnIia0Oo8HPoIjJHOXDrm2VHCx4J3tldq'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.158451", "end": "2021-02-11 06:06:58.302198", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-Y7sQo6iFEup7aLZOCZRt0qATIEk8C2RxnIia0Oo8HPoIjJHOXDrm2VHCx4J3tldq"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 06:06:58.143747", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.153641 s, 751 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.153641 s, 751 MB/s"], "stdout": "", "stdout_lines": []} 2021-02-11T06:06:58.363 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:58 +0000 (0:00:00.073) 0:03:08.931 ***** 2021-02-11T06:06:58.369 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:58.370 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:58.392 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-11T06:06:58.394 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:58.427 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:58 +0000 (0:00:00.064) 0:03:08.996 ***** 2021-02-11T06:06:58.465 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-11T06:06:58.466 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:58.500 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:58 +0000 (0:00:00.072) 0:03:09.068 ***** 2021-02-11T06:06:58.527 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2021-02-11T06:06:58.528 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:58.562 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:58 +0000 (0:00:00.061) 0:03:09.130 ***** 2021-02-11T06:06:58.589 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2021-02-11T06:06:58.590 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:58.624 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:58 +0000 (0:00:00.062) 0:03:09.193 ***** 2021-02-11T06:06:58.641 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-02-11T06:06:58.641 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-Y7sQo6iFEup7aLZOCZRt0qATIEk8C2Rxx22ulApDMD94HUbQi42OEPVJ6b30kOiP'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.161472", "end": "2021-02-11 06:06:58.607740", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-Y7sQo6iFEup7aLZOCZRt0qATIEk8C2Rxx22ulApDMD94HUbQi42OEPVJ6b30kOiP"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 06:06:58.446268", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.156779 s, 736 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.156779 s, 736 MB/s"], "stdout": "", "stdout_lines": []} 2021-02-11T06:06:58.643 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4W2W8', 'wwn-0x5000c50087b8cd81'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '602a9f0b-ccf6-4d5d-877a-542e0952c07a', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4W2W8-part1', 'wwn-0x5000c50087b8cd81-part1'], 'uuids': ['602a9f0b-ccf6-4d5d-877a-542e0952c07a']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c50087b8cd81', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W4W2W8', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4W2W8", "wwn-0x5000c50087b8cd81"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4W2W8-part1", "wwn-0x5000c50087b8cd81-part1"], "labels": [], "masters": [], "uuids": ["602a9f0b-ccf6-4d5d-877a-542e0952c07a"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "602a9f0b-ccf6-4d5d-877a-542e0952c07a"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4W2W8", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50087b8cd81"}}, "skip_reason": 2021-02-11T06:06:58.644 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2021-02-11T06:06:58.655 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:58.689 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:58 +0000 (0:00:00.065) 0:03:09.258 ***** 2021-02-11T06:06:58.800 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-02-11T06:06:58.800 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-HvUORG-0471-0h9g-pg8G-IPNY-n25z-2M7NWP', 'nvme-INTEL_SSDPEDMD400G4_CVFT534300NV400BGN', 'nvme-nvme.8086-435646543533343330304e5634303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.050444", "end": "2021-02-11 06:06:58.778712", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-HvUORG-0471-0h9g-pg8G-IPNY-n25z-2M7NWP", "nvme-INTEL_SSDPEDMD400G4_CVFT534300NV400BGN", "nvme-nvme.8086-435646543533343330304e5634303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 06:06:58.728268", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.045876 s, 2.5 GB/s", "stde 2021-02-11T06:06:58.802 INFO:teuthology.task.ansible.out:rr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.045876 s, 2.5 GB/s"], "stdout": "", "stdout_lines": []} 2021-02-11T06:06:58.812 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:58.824 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:59.084 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-Y7sQo6iFEup7aLZOCZRt0qATIEk8C2RxJZg5vwmlNBFYyKUiRfge4l10G3ZDZytH'], 'uuids': ['d5154754-1be6-4140-89ee-f9489115f8f5']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.146140", "end": "2021-02-11 06:06:59.062309", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-Y7sQo6iFEup7aLZOCZRt0qATIEk8C2RxJZg5vwmlNBFYyKUiRfge4l10G3ZDZytH"], "labels": [], "masters": [], "uuids": ["d5154754-1be6-4140-89ee-f9489115f8f5"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 06:06:58.916169", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.142141 s, 811 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.142141 s, 811 MB/s"], "stdout": "", "stdout_lines": []} 2021-02-11T06:06:59.096 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:59.097 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:59.347 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2021-02-11T06:06:59.349 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-02-11T06:06:59.384 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:59 +0000 (0:00:00.694) 0:03:09.952 ***** 2021-02-11T06:06:59.391 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-02-11T06:06:59.392 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-Y7sQo6iFEup7aLZOCZRt0qATIEk8C2RxQaLeB7vDpb5bhEnYavqBqT3KNwC2qbQ7'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.155125", "end": "2021-02-11 06:06:59.346079", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-Y7sQo6iFEup7aLZOCZRt0qATIEk8C2RxQaLeB7vDpb5bhEnYavqBqT3KNwC2qbQ7"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 06:06:59.190954", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.150562 s, 766 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.150562 s, 766 MB/s"], "stdout": "", "stdout_lines": []} 2021-02-11T06:06:59.393 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T06:06:59.649 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-Y7sQo6iFEup7aLZOCZRt0qATIEk8C2RxBJxzfXsmi0vplctt4m52u2HtFIIPM9PI'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.155536", "end": "2021-02-11 06:06:59.619459", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-Y7sQo6iFEup7aLZOCZRt0qATIEk8C2RxBJxzfXsmi0vplctt4m52u2HtFIIPM9PI"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 06:06:59.463923", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.150871 s, 765 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.150871 s, 765 MB/s"], "stdout": "", "stdout_lines": []} 2021-02-11T06:06:59.711 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:06:59 +0000 (0:00:00.326) 0:03:10.279 ***** 2021-02-11T06:06:59.968 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-02-11T06:06:59.968 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-11T06:07:00.002 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2021-02-11T06:07:00.004 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.144375", "end": "2021-02-11 06:06:59.966834", "rc": 0, "start": "2021-02-11 06:06:59.822459", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-11T06:07:00.039 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:00 +0000 (0:00:00.328) 0:03:10.608 ***** 2021-02-11T06:07:00.214 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-02-11T06:07:00.215 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} TASK [testnode : Check for physical volumes] *********************************** changed: [smithi061.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.045944", "end": "2021-02-11 06:07:00.199250", "rc": 0, "start": "2021-02-11 06:07:00.153306", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-11T06:07:00.250 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:00 +0000 (0:00:00.210) 0:03:10.818 ***** 2021-02-11T06:07:00.277 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-02-11T06:07:00.315 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:00 +0000 (0:00:00.064) 0:03:10.883 ***** 2021-02-11T06:07:00.342 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2021-02-11T06:07:00.344 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:07:00.380 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:00 +0000 (0:00:00.064) 0:03:10.948 ***** 2021-02-11T06:07:00.407 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2021-02-11T06:07:00.409 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:07:00.447 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:00 +0000 (0:00:00.066) 0:03:11.015 ***** 2021-02-11T06:07:00.476 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-02-11T06:07:00.477 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:07:00.520 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:00 +0000 (0:00:00.072) 0:03:11.088 ***** 2021-02-11T06:07:00.546 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2021-02-11T06:07:00.548 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:07:00.583 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:00 +0000 (0:00:00.063) 0:03:11.152 ***** 2021-02-11T06:07:00.707 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-02-11T06:07:00.709 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ ok: [smithi061.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-11T06:07:00.753 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:00 +0000 (0:00:00.169) 0:03:11.321 ***** 2021-02-11T06:07:00.793 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-02-11T06:07:00.795 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'block_used': 13754885, 'uuid': '602a9f0b-ccf6-4d5d-877a-542e0952c07a', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226340223, 'size_available': 927089553408, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149546, 'block_size': 4096, 'inode_available': 60905430}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226340223, "block_size": 4096, "block_total": 240095108, "block_used": 13754885, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905430, "inode_total": 61054976, "inode_used": 149546, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927089553408, "size_total": 983429562368, "uuid": "602a9f0b-ccf6-4d5d-877a-542e0952c07a"}, "skip_reason": "Conditional result was False"} 2021-02-11T06:07:00.830 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:00 +0000 (0:00:00.077) 0:03:11.399 ***** 2021-02-11T06:07:00.858 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2021-02-11T06:07:00.859 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:07:00.893 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:00 +0000 (0:00:00.062) 0:03:11.462 ***** 2021-02-11T06:07:00.941 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-11T06:07:00.943 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:07:00.981 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:00 +0000 (0:00:00.087) 0:03:11.549 ***** 2021-02-11T06:07:00.988 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-02-11T06:07:00.989 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-11T06:07:01.029 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:01 +0000 (0:00:00.047) 0:03:11.597 ***** 2021-02-11T06:07:01.035 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-11T06:07:01.036 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:07:01.073 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:01 +0000 (0:00:00.044) 0:03:11.642 ***** 2021-02-11T06:07:01.101 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2021-02-11T06:07:01.102 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:07:01.137 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:01 +0000 (0:00:00.063) 0:03:11.705 ***** 2021-02-11T06:07:01.165 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-02-11T06:07:01.167 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-02-11T06:07:01.223 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:01 +0000 (0:00:00.085) 0:03:11.791 ***** 2021-02-11T06:07:01.232 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2021-02-11T06:07:01.232 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:07:01.288 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:01 +0000 (0:00:00.064) 0:03:11.856 ***** 2021-02-11T06:07:01.316 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2021-02-11T06:07:01.318 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:07:01.352 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:01 +0000 (0:00:00.064) 0:03:11.920 ***** 2021-02-11T06:07:01.392 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-02-11T06:07:01.394 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-11T06:07:01.510 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-11T06:07:01.630 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-11T06:07:01.704 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2021-02-11T06:07:01.706 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-02-11T06:07:01.743 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:01 +0000 (0:00:00.391) 0:03:12.312 ***** 2021-02-11T06:07:01.761 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-02-11T06:07:01.763 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-11T06:07:01.773 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-02-11T06:07:01.828 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:01 +0000 (0:00:00.084) 0:03:12.396 ***** 2021-02-11T06:07:01.858 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* 2021-02-11T06:07:01.860 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:07:01.903 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:01 +0000 (0:00:00.074) 0:03:12.471 ***** 2021-02-11T06:07:01.934 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** 2021-02-11T06:07:01.936 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:07:01.970 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:01 +0000 (0:00:00.067) 0:03:12.538 ***** 2021-02-11T06:07:02.083 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-02-11T06:07:02.085 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-11T06:07:02.095 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "size": 4096, "state": "directory", "uid": 0} 2021-02-11T06:07:02.135 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:02 +0000 (0:00:00.165) 0:03:12.703 ***** 2021-02-11T06:07:02.314 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** ok: [smithi172.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2021-02-11T06:07:02.679 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:02 +0000 (0:00:00.351) 0:03:13.055 ***** TASK [testnode : Create logical volume(s)] ************************************* changed: [smithi061.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} changed: [smithi061.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-11T06:07:02.855 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-11T06:07:02.877 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* changed: [smithi172.front.sepia.ceph.com] => {"changed": true} 2021-02-11T06:07:02.914 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:02 +0000 (0:00:00.427) 0:03:13.482 ***** 2021-02-11T06:07:03.119 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-02-11T06:07:03.121 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-11T06:07:03.159 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:03 +0000 (0:00:00.244) 0:03:13.727 ***** 2021-02-11T06:07:03.270 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2021-02-11T06:07:03.272 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-02-11T06:07:03.308 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:03 +0000 (0:00:00.149) 0:03:13.876 ***** 2021-02-11T06:07:03.314 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-02-11T06:07:03.315 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-02-11T06:07:03.351 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:03 +0000 (0:00:00.042) 0:03:13.919 ***** 2021-02-11T06:07:03.357 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd] ****************************************** 2021-02-11T06:07:03.358 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:07:03.395 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:03 +0000 (0:00:00.043) 0:03:13.963 ***** 2021-02-11T06:07:03.422 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd conf file] ******************************** 2021-02-11T06:07:03.423 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:07:03.457 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:03 +0000 (0:00:00.061) 0:03:14.025 ***** 2021-02-11T06:07:03.484 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:07:03.519 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:03 +0000 (0:00:00.061) 0:03:14.087 ***** 2021-02-11T06:07:03.525 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-02-11T06:07:03.526 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-11T06:07:03.548 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-02-11T06:07:03.550 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:07:03.584 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:03 +0000 (0:00:00.065) 0:03:14.152 ***** 2021-02-11T06:07:03.612 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-02-11T06:07:03.614 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-11T06:07:03.728 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-11T06:07:03.750 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2021-02-11T06:07:03.751 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 1000} 2021-02-11T06:07:03.788 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:03 +0000 (0:00:00.204) 0:03:14.357 ***** 2021-02-11T06:07:03.837 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-02-11T06:07:03.838 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-11T06:07:03.848 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-02-11T06:07:03.887 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:03 +0000 (0:00:00.098) 0:03:14.455 ***** 2021-02-11T06:07:03.914 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* 2021-02-11T06:07:03.916 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:07:03.949 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:03 +0000 (0:00:00.061) 0:03:14.517 ***** 2021-02-11T06:07:03.955 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2021-02-11T06:07:03.956 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 0} 2021-02-11T06:07:03.992 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:03 +0000 (0:00:00.042) 0:03:14.560 ***** 2021-02-11T06:07:03.999 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** 2021-02-11T06:07:04.000 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:07:04.037 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:04 +0000 (0:00:00.044) 0:03:14.605 ***** 2021-02-11T06:07:04.180 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2021-02-11T06:07:04.181 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "size": 4096, "state": "directory", "uid": 0} 2021-02-11T06:07:04.215 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:04 +0000 (0:00:00.177) 0:03:14.783 ***** 2021-02-11T06:07:04.252 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2021-02-11T06:07:04.254 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2021-02-11T06:07:04.287 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:04 +0000 (0:00:00.072) 0:03:14.856 ***** 2021-02-11T06:07:04.346 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2021-02-11T06:07:04.348 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "gid": 1238, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 1000} 2021-02-11T06:07:04.383 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:04 +0000 (0:00:00.095) 0:03:14.952 ***** 2021-02-11T06:07:04.524 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2021-02-11T06:07:04.526 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => {"changed": true} 2021-02-11T06:07:04.561 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:04 +0000 (0:00:00.177) 0:03:15.129 ***** 2021-02-11T06:07:04.712 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2021-02-11T06:07:04.714 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/root/.cpan/CPAN/MyConfig.pm", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 0} 2021-02-11T06:07:04.749 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:04 +0000 (0:00:00.187) 0:03:15.317 ***** 2021-02-11T06:07:04.776 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2021-02-11T06:07:04.777 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-02-11T06:07:04.812 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:04 +0000 (0:00:00.063) 0:03:15.380 ***** 2021-02-11T06:07:04.839 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd] ****************************************** 2021-02-11T06:07:04.840 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:07:04.874 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:04 +0000 (0:00:00.062) 0:03:15.442 ***** 2021-02-11T06:07:04.911 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd conf file] ******************************** 2021-02-11T06:07:04.912 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:07:04.951 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:04 +0000 (0:00:00.077) 0:03:15.520 ***** 2021-02-11T06:07:05.003 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-02-11T06:07:05.004 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:07:05.038 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:05 +0000 (0:00:00.086) 0:03:15.607 ***** 2021-02-11T06:07:05.075 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-02-11T06:07:05.077 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:07:05.112 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:05 +0000 (0:00:00.073) 0:03:15.680 ***** 2021-02-11T06:07:05.253 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2021-02-11T06:07:05.254 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 1000} 2021-02-11T06:07:05.288 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:05 +0000 (0:00:00.176) 0:03:15.857 ***** 2021-02-11T06:07:05.429 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2021-02-11T06:07:05.430 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 0} 2021-02-11T06:07:05.464 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:05 +0000 (0:00:00.175) 0:03:16.033 ***** 2021-02-11T06:07:05.649 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2021-02-11T06:07:05.651 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"cache_update_time": 1613023222, "cache_updated": false, "changed": false} 2021-02-11T06:07:05.687 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:05 +0000 (0:00:00.222) 0:03:16.255 ***** 2021-02-11T06:07:05.798 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2021-02-11T06:07:05.800 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "gid": 1238, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 1000} 2021-02-11T06:07:05.837 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:05 +0000 (0:00:00.149) 0:03:16.405 ***** 2021-02-11T06:07:05.867 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2021-02-11T06:07:05.869 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "cmd": ["perldoc", "-l", "Amazon::S3"], "delta": "0:00:00.047053", "end": "2021-02-11 06:07:05.851871", "rc": 0, "start": "2021-02-11 06:07:05.804818", "stderr": "", "stderr_lines": [], "stdout": "/usr/local/share/perl/5.26.1/Amazon/S3.pm", "stdout_lines": ["/usr/local/share/perl/5.26.1/Amazon/S3.pm"]} 2021-02-11T06:07:05.905 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:05 +0000 (0:00:00.068) 0:03:16.474 ***** 2021-02-11T06:07:05.934 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2021-02-11T06:07:05.935 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:07:05.980 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:05 +0000 (0:00:00.074) 0:03:16.548 ***** 2021-02-11T06:07:06.008 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2021-02-11T06:07:06.010 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:07:06.044 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:06 +0000 (0:00:00.064) 0:03:16.613 ***** 2021-02-11T06:07:06.156 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2021-02-11T06:07:06.157 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/root/.cpan/CPAN/MyConfig.pm", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 0} 2021-02-11T06:07:06.192 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:06 +0000 (0:00:00.147) 0:03:16.761 ***** 2021-02-11T06:07:06.939 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2021-02-11T06:07:06.941 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"cache_update_time": 1613023222, "cache_updated": false, "changed": false} 2021-02-11T06:07:06.979 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:06 +0000 (0:00:00.786) 0:03:17.547 ***** 2021-02-11T06:07:07.006 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:07:07.042 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:07 +0000 (0:00:00.063) 0:03:17.611 ***** 2021-02-11T06:07:07.126 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2021-02-11T06:07:07.127 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"cache_update_time": 1613023167, "cache_updated": false, "changed": false} 2021-02-11T06:07:07.162 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:07 +0000 (0:00:00.119) 0:03:17.731 ***** 2021-02-11T06:07:07.346 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2021-02-11T06:07:07.347 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "cmd": ["perldoc", "-l", "Amazon::S3"], "delta": "0:00:00.046611", "end": "2021-02-11 06:07:07.322389", "rc": 0, "start": "2021-02-11 06:07:07.275778", "stderr": "", "stderr_lines": [], "stdout": "/usr/local/share/perl/5.26.1/Amazon/S3.pm", "stdout_lines": ["/usr/local/share/perl/5.26.1/Amazon/S3.pm"]} 2021-02-11T06:07:07.384 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:07 +0000 (0:00:00.221) 0:03:17.952 ***** 2021-02-11T06:07:07.390 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2021-02-11T06:07:07.391 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "size": 2767, "state": "file", "uid": 0} 2021-02-11T06:07:07.429 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:07 +0000 (0:00:00.045) 0:03:17.998 ***** 2021-02-11T06:07:07.436 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2021-02-11T06:07:07.437 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:07:07.473 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:07 +0000 (0:00:00.043) 0:03:18.041 ***** 2021-02-11T06:07:07.480 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2021-02-11T06:07:07.481 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:07:07.518 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:07 +0000 (0:00:00.045) 0:03:18.086 ***** 2021-02-11T06:07:07.524 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2021-02-11T06:07:07.524 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:07:07.562 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:07 +0000 (0:00:00.043) 0:03:18.130 ***** 2021-02-11T06:07:07.729 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2021-02-11T06:07:07.730 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "ntp", "state": "started", "status": {"ActiveEnterTimestamp": "Thu 2021-02-11 06:03:27 UTC", "ActiveEnterTimestampMonotonic": "24940232", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "network.target systemd-tmpfiles-setup.service system.slice systemd-journald.socket sysinit.target -.mount basic.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2021-02-11 06:03:21 UTC", "AssertTimestampMonotonic": "18368247", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2021-02-11 06:03:21 UTC", "ConditionTimestampMonotonic": "18368247", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target systemd-timesyncd.service", "ControlGroup": "/system.slice/ntp.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "Documen 2021-02-11T06:07:07.731 INFO:teuthology.task.ansible.out:tation": "man:ntpd(8)", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1854", "ExecMainStartTimestamp": "Thu 2021-02-11 06:03:27 UTC", "ExecMainStartTimestampMonotonic": "24940199", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/ntp.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "ntp.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2021-02-11 06:03:21 UTC", "InactiveExitTimestampMonotonic": "18369058", "InvocationID": "a2f0f005861148179b8c273db6a760a9", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128029", "LimitNPROCSoft": "128029", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "Limi 2021-02-11T06:07:07.733 INFO:teuthology.task.ansible.out:tRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "128029", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1854", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "ntp.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target -.mount", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2021-02-11 06:03:27 UTC", "S 2021-02-11T06:07:07.734 INFO:teuthology.task.ansible.out:tateChangeTimestampMonotonic": "24940232", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "2", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2021-02-11 06:03:27 UTC", "WatchdogTimestampMonotonic": "24940229", "WatchdogUSec": "0"}} 2021-02-11T06:07:07.767 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:07 +0000 (0:00:00.205) 0:03:18.336 ***** 2021-02-11T06:07:07.795 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-11T06:07:07.796 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-02-11T06:07:07.829 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:07 +0000 (0:00:00.061) 0:03:18.397 ***** 2021-02-11T06:07:07.864 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-11T06:07:07.865 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:07:07.897 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:07 +0000 (0:00:00.067) 0:03:18.465 ***** 2021-02-11T06:07:07.924 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2021-02-11T06:07:07.925 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:07:07.958 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:07 +0000 (0:00:00.060) 0:03:18.526 ***** 2021-02-11T06:07:08.434 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2021-02-11T06:07:18.178 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"cache_update_time": 1613023167, "cache_updated": false, "changed": false} Thursday 11 February 2021 06:07:08 +0000 (0:00:00.512) 0:03:19.039 ***** TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:07:08 +0000 (0:00:00.063) 0:03:19.103 ***** TASK [testnode : Create the ntp.conf file.] ************************************ ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "size": 2767, "state": "file", "uid": 0} TASK [testnode : Install python-pip on apt based systems.] ********************* ok: [smithi172.front.sepia.ceph.com] => {"cache_update_time": 1613023222, "cache_updated": false, "changed": false} Thursday 11 February 2021 06:07:08 +0000 (0:00:00.335) 0:03:19.438 ***** Thursday 11 February 2021 06:07:08 +0000 (0:00:00.031) 0:03:19.470 ***** TASK [testnode : Create the chrony.conf file] ********************************** skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:07:08 +0000 (0:00:00.040) 0:03:19.510 ***** TASK [testnode : Install python-pip on zypper based systems.] ****************** skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:07:08 +0000 (0:00:00.042) 0:03:19.553 ***** TASK [testnode : Create the .pip directory for the teuthology user.] *********** ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "size": 4 2021-02-11T06:07:18.179 INFO:teuthology.task.ansible.out:096, "state": "directory", "uid": 1000} Thursday 11 February 2021 06:07:09 +0000 (0:00:00.165) 0:03:19.718 ***** TASK [testnode : Make sure ntpd is running.] *********************************** ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "ntp", "state": "started", "status": {"ActiveEnterTimestamp": "Thu 2021-02-11 06:03:16 UTC", "ActiveEnterTimestampMonotonic": "26127285", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-tmpfiles-setup.service basic.target -.mount network.target systemd-journald.socket sysinit.target system.slice", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2021-02-11 06:03:10 UTC", "AssertTimestampMonotonic": "19640033", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2021-02-11 06:03:10 UTC", "ConditionTimestampMonotonic": "19640033", "ConfigurationDirectoryMode": "0755", "Conflicts": "systemd-timesyncd.se 2021-02-11T06:07:18.181 INFO:teuthology.task.ansible.out:rvice shutdown.target", "ControlGroup": "/system.slice/ntp.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "Documentation": "man:ntpd(8)", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1795", "ExecMainStartTimestamp": "Thu 2021-02-11 06:03:16 UTC", "ExecMainStartTimestampMonotonic": "26127272", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/ntp.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "ntp.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2021-02-11 06:03:10 UTC", "InactiveExitTimestampMonotonic": "19640706", "InvocationID": "a3d60dd496a742139639607b32576b39", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": " 2021-02-11T06:07:18.182 INFO:teuthology.task.ansible.out:0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128029", "LimitNPROCSoft": "128029", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "128029", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1795", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "ntp.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target -.mount", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterva 2021-02-11T06:07:18.183 INFO:teuthology.task.ansible.out:lUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2021-02-11 06:03:16 UTC", "StateChangeTimestampMonotonic": "26127285", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "2", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2021-02-11 06:03:16 UTC", "WatchdogTimestampMonotonic": "26127284", "WatchdogUSec": "0"}} Thursday 11 February 2021 06:07:09 +0000 (0:00:00.062) 0:03:19.781 ***** TASK [testnode : set_fact] ***************************************************** ok: [smithi061.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} Thursday 11 February 2021 06:07:09 +0000 (0:00:00.061) 0:03:19.842 ***** TASK [testnode : set_fact] ***************************************************** skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:07:09 +0000 (0:00:00.064) 0:03:19.907 ***** TASK [testnode : Install python-pip on rpm based systems.] ********************* skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:07:09 +0000 (0:00:00.061) 0:03:19.968 ***** TASK [testnode : Create pip.conf and 2021-02-11T06:07:18.185 INFO:teuthology.task.ansible.out:configure it to use our mirror] *********** ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "size": 73, "state": "file", "uid": 1000} Thursday 11 February 2021 06:07:09 +0000 (0:00:00.103) 0:03:20.072 ***** TASK [testnode : Install python-pip on apt based systems.] ********************* ok: [smithi061.front.sepia.ceph.com] => {"cache_update_time": 1613023167, "cache_updated": false, "changed": false} Thursday 11 February 2021 06:07:10 +0000 (0:00:00.824) 0:03:20.896 ***** TASK [testnode : Install python-pip on zypper based systems.] ****************** skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:07:10 +0000 (0:00:00.061) 0:03:20.958 ***** TASK [testnode : Install packages via pip] ************************************* ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "cmd": ["/usr/bin/pip", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "", "stderr_lines": [], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages\nRequirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages", "Requirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} Thursday 11 F 2021-02-11T06:07:18.186 INFO:teuthology.task.ansible.out:ebruary 2021 06:07:10 +0000 (0:00:00.052) 0:03:21.010 ***** TASK [testnode : Create the .pip directory for the teuthology user.] *********** ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "size": 4096, "state": "directory", "uid": 1000} Thursday 11 February 2021 06:07:10 +0000 (0:00:00.105) 0:03:21.116 ***** TASK [testnode : Create pip.conf and configure it to use our mirror] *********** ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "size": 73, "state": "file", "uid": 1000} Thursday 11 February 2021 06:07:10 +0000 (0:00:00.354) 0:03:21.471 ***** TASK [testnode : Purge resolvconf] ********************************************* ok: [smithi172.front.sepia.ceph.com] => {"changed": false} Thursday 11 February 2021 06:07:10 +0000 (0:00:00.074) 0:03:21.545 ***** TASK [testnode : Set interface] ************************************************ ok: [smithi172.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} Thursday 11 February 2021 06:07:11 +0000 (0:00:00.061) 0:03:21.607 ***** TASK [testnode : Check for presence of /etc/network/interfaces] **************** ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1613023072.448325, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "ctime": 1613023072.444325, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 746, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/plain", "mode 2021-02-11T06:07:18.188 INFO:teuthology.task.ansible.out:": "0644", "mtime": 1613023072.444325, "nlink": 1, "path": "/etc/network/interfaces", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 71, "uid": 0, "version": "3375002756", "wgrp": false, "woth": false, "writeable": 2021-02-11T06:07:18.188 INFO:teuthology.task.ansible.out:true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} 2021-02-11T06:07:18.189 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2021-02-11T06:07:18.189 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "cmd": ["/usr/bin/pip", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "", "stderr_lines": [], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages\nRequirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages", "Requirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2021-02-11T06:07:18.228 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:18 +0000 (0:00:07.189) 0:03:28.797 ***** 2021-02-11T06:07:18.296 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:18 +0000 (0:00:00.067) 0:03:28.864 ***** 2021-02-11T06:07:18.563 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2021-02-11T06:07:18.564 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "msg": ""} 2021-02-11T06:07:18.598 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:18 +0000 (0:00:00.302) 0:03:29.167 ***** 2021-02-11T06:07:18.626 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2021-02-11T06:07:18.627 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-02-11T06:07:18.661 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:18 +0000 (0:00:00.062) 0:03:29.229 ***** 2021-02-11T06:07:18.688 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2021-02-11T06:07:18.689 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:07:18.723 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:18 +0000 (0:00:00.062) 0:03:29.292 ***** 2021-02-11T06:07:18.792 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2021-02-11T06:07:18.794 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"changed": false} 2021-02-11T06:07:18.829 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:18 +0000 (0:00:00.105) 0:03:29.397 ***** 2021-02-11T06:07:18.847 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2021-02-11T06:07:18.847 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-02-11T06:07:18.882 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:18 +0000 (0:00:00.053) 0:03:29.451 ***** 2021-02-11T06:07:18.888 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2021-02-11T06:07:18.889 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2021-02-11T06:07:18.926 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:18 +0000 (0:00:00.043) 0:03:29.494 ***** 2021-02-11T06:07:19.006 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2021-02-11T06:07:19.007 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-02-11T06:07:19.048 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:19 +0000 (0:00:00.121) 0:03:29.616 ***** 2021-02-11T06:07:19.065 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2021-02-11T06:07:19.066 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1613023064.0003626, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "ctime": 1613023063.9963627, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 746, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/plain", "mode": "0644", "mtime": 1613023063.9963627, "nlink": 1, "path": "/etc/network/interfaces", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 71, "uid": 0, "version": "3375002756", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} 2021-02-11T06:07:19.101 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:19 +0000 (0:00:00.053) 0:03:29.669 ***** 2021-02-11T06:07:19.107 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2021-02-11T06:07:19.107 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:07:19.145 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:19 +0000 (0:00:00.044) 0:03:29.713 ***** 2021-02-11T06:07:19.183 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2021-02-11T06:07:19.184 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2021-02-11T06:07:19.194 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2021-02-11T06:07:19.195 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2021-02-11T06:07:19.196 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} 2021-02-11T06:07:19.231 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:19 +0000 (0:00:00.085) 0:03:29.799 ***** 2021-02-11T06:07:19.238 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2021-02-11T06:07:19.238 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "msg": ""} 2021-02-11T06:07:19.277 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:19 +0000 (0:00:00.046) 0:03:29.845 ***** 2021-02-11T06:07:19.285 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2021-02-11T06:07:19.286 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi172.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-02-11T06:07:19.286 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-02-11T06:07:19.288 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-02-11T06:07:19.298 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-02-11T06:07:19.334 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:19 +0000 (0:00:00.057) 0:03:29.902 ***** 2021-02-11T06:07:19.340 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2021-02-11T06:07:19.340 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-02-11T06:07:19.376 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:19 +0000 (0:00:00.042) 0:03:29.944 ***** 2021-02-11T06:07:19.382 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-11T06:07:19.382 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:07:19.421 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:19 +0000 (0:00:00.045) 0:03:29.990 ***** 2021-02-11T06:07:19.428 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2021-02-11T06:07:19.430 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:07:19.465 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:19 +0000 (0:00:00.043) 0:03:30.033 ***** 2021-02-11T06:07:19.473 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2021-02-11T06:07:19.475 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:07:19.510 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:19 +0000 (0:00:00.045) 0:03:30.079 ***** 2021-02-11T06:07:19.538 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2021-02-11T06:07:19.539 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:07:19.575 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:19 +0000 (0:00:00.064) 0:03:30.143 ***** 2021-02-11T06:07:19.592 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2021-02-11T06:07:19.594 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-02-11T06:07:19.628 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:19 +0000 (0:00:00.052) 0:03:30.196 ***** 2021-02-11T06:07:19.634 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2021-02-11T06:07:19.634 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:07:19.677 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:19 +0000 (0:00:00.049) 0:03:30.245 ***** 2021-02-11T06:07:19.704 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2021-02-11T06:07:19.706 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:07:19.742 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:19 +0000 (0:00:00.064) 0:03:30.310 ***** 2021-02-11T06:07:19.759 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2021-02-11T06:07:19.761 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-02-11T06:07:19.795 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:19 +0000 (0:00:00.053) 0:03:30.363 ***** 2021-02-11T06:07:19.801 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-02-11T06:07:19.803 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:07:19.839 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:19 +0000 (0:00:00.043) 0:03:30.407 ***** 2021-02-11T06:07:19.845 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2021-02-11T06:07:19.846 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:07:19.882 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:19 +0000 (0:00:00.042) 0:03:30.450 ***** 2021-02-11T06:07:19.889 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2021-02-11T06:07:19.890 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:07:19.929 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:19 +0000 (0:00:00.046) 0:03:30.497 ***** 2021-02-11T06:07:19.934 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2021-02-11T06:07:19.936 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2021-02-11T06:07:19.937 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2021-02-11T06:07:19.973 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:19 +0000 (0:00:00.043) 0:03:30.541 ***** 2021-02-11T06:07:19.978 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-02-11T06:07:19.980 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:07:20.017 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:20 +0000 (0:00:00.044) 0:03:30.586 ***** 2021-02-11T06:07:20.024 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2021-02-11T06:07:20.025 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-02-11T06:07:20.026 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-02-11T06:07:20.028 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-02-11T06:07:20.038 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-02-11T06:07:20.077 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:20 +0000 (0:00:00.059) 0:03:30.646 ***** 2021-02-11T06:07:20.105 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-11T06:07:20.107 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:07:20.143 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:20 +0000 (0:00:00.065) 0:03:30.711 ***** 2021-02-11T06:07:20.169 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2021-02-11T06:07:20.170 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** ok: [smithi172.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.011000", "end": "2021-02-11 06:07:20.142976", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-02-11 06:07:20.131976", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-11T06:07:20.205 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:20 +0000 (0:00:00.062) 0:03:30.774 ***** 2021-02-11T06:07:20.233 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2021-02-11T06:07:20.234 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:07:20.270 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:20 +0000 (0:00:00.064) 0:03:30.838 ***** 2021-02-11T06:07:20.299 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-02-11T06:07:20.300 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.006032", "end": "2021-02-11 06:07:20.263972", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-02-11 06:07:20.257940", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:07:20.345 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:20 +0000 (0:00:00.074) 0:03:30.913 ***** 2021-02-11T06:07:23.079 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:07:20 +0000 (0:00:00.068) 0:03:30.981 ***** TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** ok: [smithi172.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.004678", "end": "2021-02-11 06:07:20.388384", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-02-11 06:07:20.383706", "stderr": "find: '/etc/ceph': No such file or directory", "stderr_lines": ["find: '/etc/ceph': No such file or directory"], "stdout": "", "stdout_lines": []} TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:07:20 +0000 (0:00:00.067) 0:03:31.049 ***** TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:07:20 +0000 (0:00:00.062) 0:03:31.112 ***** TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** ok: [smithi172.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.035442", "end": "2021-02-11 06:07:20.544068", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-02-11 06: 2021-02-11T06:07:23.080 INFO:teuthology.task.ansible.out:07:20.508626", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} Thursday 11 February 2021 06:07:20 +0000 (0:00:00.063) 0:03:31.176 ***** Thursday 11 February 2021 06:07:20 +0000 (0:00:00.033) 0:03:31.210 ***** TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** Thursday 11 February 2021 06:07:20 +0000 (0:00:00.061) 0:03:31.272 ***** TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** ok: [smithi061.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.010861", "end": "2021-02-11 06:07:20.733634", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-02-11 06:07:20.722773", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} TASK [testnode : Touch /ceph-qa-ready] ***************************************** changed: [smithi172.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 0, "state": "file", "uid": 0} TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** ok: [smithi061.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.006057", "end": "2021-02-11 06:07:20.856060", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-02-11 06:07:20.850003", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} ok: [smithi061.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.005543", "end": "2021-02-11 06:07:20.974061", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-02-11 06:07:20.968518", "stderr": " 2021-02-11T06:07:23.081 INFO:teuthology.task.ansible.out:find: '/etc/ceph': No such file or directory", "stderr_lines": ["find: '/etc/ceph': No such file or directory"], "stdout": "", "stdout_lines": []} ok: [smithi061.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.012174", "end": "2021-02-11 06:07:21.096116", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-02-11 06:07:21.083942", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} Thursday 11 February 2021 06:07:21 +0000 (0:00:00.461) 0:03:31.733 ***** TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** Thursday 11 February 2021 06:07:21 +0000 (0:00:00.061) 0:03:31.795 ***** TASK [testnode : Touch /ceph-qa-ready] ***************************************** changed: [smithi061.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 0, "state": "file", "uid": 0} PLAY [testnodes,senta,vossi,jenkins_slaves] ************************************ Thursday 11 February 2021 06:07:21 +0000 (0:00:00.269) 0:03:32.065 ***** Thursday 11 February 2021 06:07:21 +0000 (0:00:00.035) 0:03:32.100 ***** TASK [Gathering Facts] ********************************************************* ok: [smithi061.front.sepia.ceph.com] ok: [smithi172.front.sepia.ceph.com] Thursday 11 February 2021 06:07:22 +0000 (0:00:00.808) 0:03:32.909 ***** TASK [container-host : set_fact] *********************************************** ok: [smithi061.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "apt"}, "changed": false} Thursday 11 February 2021 06:07:22 +0000 (0:00:00.079) 0:03:32.989 ***** Thursday 11 February 2021 06:07:22 +0000 (0:00:00.035) 0:03:33.024 ***** TASK [container-host : set_fact] *********************************************** skipping: [smithi061.fron 2021-02-11T06:07:23.083 INFO:teuthology.task.ansible.out:t.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:07:22 +0000 (0:00:00.043) 0:03:33.068 ***** TASK [container-host : set_fact] *********************************************** ok: [smithi172.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "apt"}, "changed": false} Thursday 11 February 2021 06:07:22 +0000 (0:00:00.046) 0:03:33.115 ***** TASK [container-host : Including distro specific variables] ******************** ok: [smithi061.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml) => {"ansible_facts": {"container_packages": ["docker.io"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml"} Thursday 11 February 2021 06:07:22 +0000 (0:00:00.046) 0:03:33.161 ***** TASK [container-host : set_fact] *********************************************** skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 06:07:22 +0000 (0:00:00.045) 0:03:33.206 ***** TASK [container-host : Including distro specific variables] ******************** ok: [smithi172.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml) => {"ansible_facts": {"container_packages": ["docker.io"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml"} Thursd 2021-02-11T06:07:23.084 INFO:teuthology.task.ansible.out:ay 11 February 2021 06:07:22 +0000 (0:00:00.074) 0:03:33.281 ***** 2021-02-11T06:07:23.524 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** 2021-02-11T06:07:23.525 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"cache_update_time": 1613023167, "cache_updated": false, "changed": false} 2021-02-11T06:07:23.564 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:23 +0000 (0:00:00.850) 0:03:34.132 ***** 2021-02-11T06:07:23.591 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-02-11T06:07:23.592 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:07:23.629 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:23 +0000 (0:00:00.064) 0:03:34.197 ***** 2021-02-11T06:07:23.666 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-02-11T06:07:23.668 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/docker/daemon.json"}, "changed": false} 2021-02-11T06:07:23.709 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:23 +0000 (0:00:00.080) 0:03:34.278 ***** 2021-02-11T06:07:23.716 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** 2021-02-11T06:07:23.716 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"cache_update_time": 1613023222, "cache_updated": false, "changed": false} 2021-02-11T06:07:23.760 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:23 +0000 (0:00:00.050) 0:03:34.328 ***** 2021-02-11T06:07:23.787 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-02-11T06:07:23.789 INFO:teuthology.task.ansible.out:skipping: [smithi172.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T06:07:23.826 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:23 +0000 (0:00:00.066) 0:03:34.395 ***** 2021-02-11T06:07:23.864 INFO:teuthology.task.ansible.out: TASK [container-host : Create container_mirror_cert_paths] ********************* 2021-02-11T06:07:23.866 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "size": 4096, "state": "directory", "uid": 0} TASK [container-host : set_fact] *********************************************** ok: [smithi172.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/docker/daemon.json"}, "changed": false} 2021-02-11T06:07:23.904 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:23 +0000 (0:00:00.077) 0:03:34.472 ***** 2021-02-11T06:07:23.972 INFO:teuthology.task.ansible.out: TASK [container-host : Create container_mirror_cert_paths] ********************* 2021-02-11T06:07:23.974 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "size": 4096, "state": "directory", "uid": 0} 2021-02-11T06:07:24.013 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:24 +0000 (0:00:00.109) 0:03:34.582 ***** 2021-02-11T06:07:24.061 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "size": 4096, "state": "directory", "uid": 0} 2021-02-11T06:07:24.166 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "size": 4096, "state": "directory", "uid": 0} 2021-02-11T06:07:24.207 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:24 +0000 (0:00:00.193) 0:03:34.776 ***** 2021-02-11T06:07:24.390 INFO:teuthology.task.ansible.out: TASK [container-host : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert] *** 2021-02-11T06:07:24.392 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1613023644.0513783-109792381105749/source", "state": "file", "uid": 0} 2021-02-11T06:07:24.559 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1613023644.243992-260312088141345/source", "state": "file", "uid": 0} 2021-02-11T06:07:24.684 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1613023644.3946774-92823520904702/source", "state": "file", "uid": 0} 2021-02-11T06:07:24.723 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:24 +0000 (0:00:00.515) 0:03:35.291 ***** 2021-02-11T06:07:24.844 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1613023644.5641787-239770275252957/source", "state": "file", "uid": 0} 2021-02-11T06:07:24.885 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:24 +0000 (0:00:00.161) 0:03:35.453 ***** 2021-02-11T06:07:28.321 INFO:teuthology.task.ansible.out: TASK [container-host : Install registries-conf-ctl] **************************** 2021-02-11T06:07:28.322 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => {"changed": true, "cmd": ["/usr/bin/pip2", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "", "stderr_lines": [], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-Z85LJF-build\nCollecting docopt (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz\nCollecting toml (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl\nCollecting typing (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/3b/c0/e44213fcb799eac02881e2485724ba5b0914600bc9df6ed922e364fdc059/typing-3.7.4.3-py2-none-any.whl\nBuilding wheels for collected packages: docopt\n Running setup.py bdist_wheel for docopt: started\n Running setup.py bdist_wheel for docopt: finished with status 'done'\n Stored in directory: /root/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e\nSuccessfully built docopt\nInstalling collected packages: docopt, toml, typing, registries-conf-ctl\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2 typing-3.7.4.3\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-Z85LJF-build", "Collecting docopt (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packag 2021-02-11T06:07:28.323 INFO:teuthology.task.ansible.out:es/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz", "Collecting toml (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl", "Collecting typing (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/3b/c0/e44213fcb799eac02881e2485724ba5b0914600bc9df6ed922e364fdc059/typing-3.7.4.3-py2-none-any.whl", "Building wheels for collected packages: docopt", " Running setup.py bdist_wheel for docopt: started", " Running setup.py bdist_wheel for docopt: finished with status 'done'", " Stored in directory: /root/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e", "Successfully built docopt", "Installing collected packages: docopt, toml, typing, registries-conf-ctl", " Running setup.py install for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2 typing-3.7.4.3"], "version": null, "virtualenv": null} 2021-02-11T06:07:28.361 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:28 +0000 (0:00:03.476) 0:03:38.930 ***** 2021-02-11T06:07:28.380 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => {"changed": true, "cmd": ["/usr/bin/pip2", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "", "stderr_lines": [], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-EdvDyp-build\nCollecting docopt (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz\nCollecting toml (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl\nCollecting typing (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/3b/c0/e44213fcb799eac02881e2485724ba5b0914600bc9df6ed922e364fdc059/typing-3.7.4.3-py2-none-any.whl\nBuilding wheels for collected packages: docopt\n Running setup.py bdist_wheel for docopt: started\n Running setup.py bdist_wheel for docopt: finished with status 'done'\n Stored in directory: /root/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e\nSuccessfully built docopt\nInstalling collected packages: docopt, toml, typing, registries-conf-ctl\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2 typing-3.7.4.3\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-EdvDyp-build", "Collecting docopt (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packag 2021-02-11T06:07:28.382 INFO:teuthology.task.ansible.out:es/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz", "Collecting toml (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl", "Collecting typing (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/3b/c0/e44213fcb799eac02881e2485724ba5b0914600bc9df6ed922e364fdc059/typing-3.7.4.3-py2-none-any.whl", "Building wheels for collected packages: docopt", " Running setup.py bdist_wheel for docopt: started", " Running setup.py bdist_wheel for docopt: finished with status 'done'", " Stored in directory: /root/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e", "Successfully built docopt", "Installing collected packages: docopt, toml, typing, registries-conf-ctl", " Running setup.py install for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2 typing-3.7.4.3"], "version": null, "virtualenv": null} 2021-02-11T06:07:28.422 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:28 +0000 (0:00:00.060) 0:03:38.990 ***** 2021-02-11T06:07:28.512 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* 2021-02-11T06:07:28.513 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-02-11T06:07:28.553 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:28 +0000 (0:00:00.130) 0:03:39.121 ***** 2021-02-11T06:07:28.559 INFO:teuthology.task.ansible.out:ok: [smithi172.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-02-11T06:07:28.598 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:28 +0000 (0:00:00.045) 0:03:39.166 ***** 2021-02-11T06:07:28.917 INFO:teuthology.task.ansible.out: TASK [container-host : Create /etc/docker/daemon.json if necessary] ************ 2021-02-11T06:07:28.919 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => {"changed": true, "checksum": "bf21a9e8fbc5a3846fb05b4fa0859e0917b2202f", "dest": "/etc/docker/daemon.json", "gid": 0, "group": "root", "md5sum": "99914b932bd37a50b983c5e7c90ae93b", "mode": "0644", "owner": "root", "size": 2, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1613023648.5865743-44535518620295/source", "state": "file", "uid": 0} changed: [smithi172.front.sepia.ceph.com] => {"changed": true, "checksum": "bf21a9e8fbc5a3846fb05b4fa0859e0917b2202f", "dest": "/etc/docker/daemon.json", "gid": 0, "group": "root", "md5sum": "99914b932bd37a50b983c5e7c90ae93b", "mode": "0644", "owner": "root", "size": 2, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1613023648.6318495-25607268995362/source", "state": "file", "uid": 0} 2021-02-11T06:07:28.956 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:28 +0000 (0:00:00.357) 0:03:39.524 ***** 2021-02-11T06:07:29.012 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:29 +0000 (0:00:00.056) 0:03:39.580 ***** 2021-02-11T06:07:29.237 INFO:teuthology.task.ansible.out: TASK [container-host : Add local docker.io registry mirror] ******************** 2021-02-11T06:07:29.239 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.120899", "end": "2021-02-11 06:07:29.218101", "rc": 0, "start": "2021-02-11 06:07:29.097202", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-11T06:07:29.276 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:29 +0000 (0:00:00.264) 0:03:39.844 ***** 2021-02-11T06:07:29.282 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.127221", "end": "2021-02-11 06:07:29.253659", "rc": 0, "start": "2021-02-11 06:07:29.126438", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-11T06:07:29.322 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 06:07:29 +0000 (0:00:00.046) 0:03:39.890 ***** 2021-02-11T06:07:32.610 INFO:teuthology.task.ansible.out: TASK [container-host : Restart docker service] ********************************* 2021-02-11T06:07:32.611 INFO:teuthology.task.ansible.out:changed: [smithi172.front.sepia.ceph.com] => {"changed": true, "name": "docker", "state": "started", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "docker.socket basic.target sysinit.target systemd-journald.socket firewalld.service network-online.target system.slice containerd.service", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ConsistsOf": "docker.socket", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "yes", "DelegateControllers": "cpu cpuacct io blkio memory devices pids", "Description": "Docker Application Container Engine", "DevicePolicy": "auto", "Documentation": "https://docs.docker.com", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMa 2021-02-11T06:07:32.612 INFO:teuthology.task.ansible.out:inStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -s HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/bin/dockerd ; argv[]=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/docker.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "docker.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "process", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "infinity", "LimitNOFILESoft": "infinity", "LimitNPROC": "infinity", "LimitNPROCSoft": "infinity", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "12802 2021-02-11T06:07:32.613 INFO:teuthology.task.ansible.out:9", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "docker.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "main", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target system.slice docker.socket", "Restart": "always", "RestartUSec": "2s", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "3", "StartLimitIntervalUSec": "1min", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2021-02-11 06:05:43 UTC", "StateChangeTimestampMonotonic": "160502838", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "S 2021-02-11T06:07:32.615 INFO:teuthology.task.ansible.out:uccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "infinity", "TimeoutStartUSec": "infinity", "TimeoutStopUSec": "infinity", "TimerSlackNSec": "50000", "Transient": "no", "TriggeredBy": "docker.socket", "Type": "notify", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "containerd.service network-online.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} changed: [smithi061.front.sepia.ceph.com] => {"changed": true, "name": "docker", "state": "started", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "systemd-journald.socket network-online.target sysinit.target containerd.service docker.socket firewalld.service system.slice basic.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease c 2021-02-11T06:07:32.616 INFO:teuthology.task.ansible.out:ap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ConsistsOf": "docker.socket", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "yes", "DelegateControllers": "cpu cpuacct io blkio memory devices pids", "Description": "Docker Application Container Engine", "DevicePolicy": "auto", "Documentation": "https://docs.docker.com", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -s HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/bin/dockerd ; argv[]=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/docker.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "docker.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "process", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "L 2021-02-11T06:07:32.617 INFO:teuthology.task.ansible.out:imitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "infinity", "LimitNOFILESoft": "infinity", "LimitNPROC": "infinity", "LimitNPROCSoft": "infinity", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "128029", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "docker.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "main", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice docker.socket sysinit.target", "Restart": "always", "RestartUSec": "2s", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": 2021-02-11T06:07:32.619 INFO:teuthology.task.ansible.out: "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "3", "StartLimitIntervalUSec": "1min", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2021-02-11 06:05:47 UTC", "StateChangeTimestampMonotonic": "177284426", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "infinity", "TimeoutStartUSec": "infinity", "TimeoutStopUSec": "infinity", "TimerSlackNSec": "50000", "Transient": "no", "TriggeredBy": "docker.socket", "Type": "notify", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "network-online.target containerd.service", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-02-11T06:07:32.767 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2021-02-11T06:07:32.773 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** skipping: no hosts matched 2021-02-11T06:07:32.780 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** 2021-02-11T06:07:32.781 INFO:teuthology.task.ansible.out:skipping: no hosts matched PLAY RECAP ********************************************************************* smithi061.front.sepia.ceph.com : ok=126 changed=38 unreachable=0 failed=0 skipped=184 rescued=0 ignored=0 smithi172.front.sepia.ceph.com : ok=126 changed=38 unreachable=0 failed=0 skipped=184 rescued=0 ignored=0 Thursday 11 February 2021 06:07:32 +0000 (0:00:03.458) 0:03:43.349 ***** =============================================================================== users : Create all admin users with sudo access. ----------------------- 25.50s testnode : Set root disk ----------------------------------------------- 13.05s users : Update authorized_keys for each user with literal keys --------- 12.43s users : Update authorized_keys using the keys repo --------------------- 11.84s testnode : Use perl to add user_xattr to the root mount options in fstab. --- 7.04s common : Add krb5 config file ------------------------------------------- 6.25s testnode : Ensure packages are not present. ----------------------------- 4.21s testnode : Install packages --------------------------------------------- 3.88s container-host : Install registries-conf-ctl ---------------------------- 3.48s container-host : Restart docker service --------------------------------- 3.46s testnode : Update apt cache. -------------------------------------------- 3.12s Gathering Facts --------------------------------------------------------- 1.95s testnode : Zap all non-root disks --------------------------------------- 1.44s users : Clone the keys repo --------------------------------------------- 1.03s testnode : Make sure apt dependencies are installed --------------------- 1.02s 2021-02-11T06:07:32.782 INFO:teuthology.task.ansible.out: testnode : Install apt keys --------------------------------------------- 1.02s testnode : Install python-apt ------------------------------------------- 0.97s testnode : Upgrade packages --------------------------------------------- 0.95s 2021-02-11T06:07:32.782 INFO:teuthology.task.ansible.out:container-host 2021-02-11T06:07:32.783 INFO:teuthology.task.ansible.out: : Install container packages ----------------------------- 0.85s common : Set the hardware clock ----------------------------------------- 0.83s 2021-02-11T06:07:33.185 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi061.front.sepia.ceph.com'), Remote(name='ubuntu@smithi172.front.sepia.ceph.com')] 2021-02-11T06:07:47.394 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-11T06:07:47.397 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-02-11T06:07:47.472 DEBUG:teuthology.orchestra.run.smithi061:> true 2021-02-11T06:07:47.679 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-11T06:07:47.680 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-02-11T06:07:47.753 DEBUG:teuthology.orchestra.run.smithi172:> true 2021-02-11T06:07:47.956 INFO:teuthology.run_tasks:Running task clock... 2021-02-11T06:07:47.976 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2021-02-11T06:07:47.976 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-02-11T06:07:47.980 DEBUG:teuthology.orchestra.run.smithi061:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-02-11T06:07:48.047 INFO:teuthology.orchestra.run.smithi061.stdout:11 Feb 06:07:48 ntpd[8414]: ntpd 4.2.8p10@1.3728-o (1): Starting 2021-02-11T06:07:48.048 INFO:teuthology.orchestra.run.smithi061.stdout:11 Feb 06:07:48 ntpd[8414]: Command line: ntpd -gq 2021-02-11T06:07:48.048 INFO:teuthology.orchestra.run.smithi061.stdout:11 Feb 06:07:48 ntpd[8414]: proto: precision = 0.127 usec (-23) 2021-02-11T06:07:48.048 INFO:teuthology.orchestra.run.smithi061.stdout:11 Feb 06:07:48 ntpd[8414]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2021-02-11T06:07:48.048 INFO:teuthology.orchestra.run.smithi061.stdout:11 Feb 06:07:48 ntpd[8414]: restrict ::: KOD does nothing without LIMITED. 2021-02-11T06:07:48.049 INFO:teuthology.orchestra.run.smithi061.stdout:11 Feb 06:07:48 ntpd[8414]: Listen and drop on 0 v6wildcard [::]:123 2021-02-11T06:07:48.049 INFO:teuthology.orchestra.run.smithi061.stdout:11 Feb 06:07:48 ntpd[8414]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2021-02-11T06:07:48.049 INFO:teuthology.orchestra.run.smithi061.stdout:11 Feb 06:07:48 ntpd[8414]: Listen normally on 2 lo 127.0.0.1:123 2021-02-11T06:07:48.049 INFO:teuthology.orchestra.run.smithi061.stdout:11 Feb 06:07:48 ntpd[8414]: Listen normally on 3 enp3s0f1 172.21.15.61:123 2021-02-11T06:07:48.049 INFO:teuthology.orchestra.run.smithi061.stdout:11 Feb 06:07:48 ntpd[8414]: Listen normally on 4 lo [::1]:123 2021-02-11T06:07:48.050 INFO:teuthology.orchestra.run.smithi061.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2021-02-11T06:07:48.050 INFO:teuthology.orchestra.run.smithi061.stderr:restrict ::: KOD does nothing without LIMITED. 2021-02-11T06:07:48.051 INFO:teuthology.orchestra.run.smithi061.stdout:11 Feb 06:07:48 ntpd[8414]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:feb6:f8b9%5]:123 2021-02-11T06:07:48.051 INFO:teuthology.orchestra.run.smithi061.stdout:11 Feb 06:07:48 ntpd[8414]: Listening on routing socket on fd #22 for interface updates 2021-02-11T06:07:49.044 INFO:teuthology.orchestra.run.smithi061.stderr:11 Feb 06:07:49 ntpd[8414]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T06:07:49.044 INFO:teuthology.orchestra.run.smithi061.stderr:11 Feb 06:07:49 ntpd[8414]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T06:07:49.044 INFO:teuthology.orchestra.run.smithi061.stderr:11 Feb 06:07:49 ntpd[8414]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T06:07:49.044 INFO:teuthology.orchestra.run.smithi061.stderr:11 Feb 06:07:49 ntpd[8414]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T06:07:49.045 INFO:teuthology.orchestra.run.smithi061.stderr:11 Feb 06:07:49 ntpd[8414]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T06:07:49.045 INFO:teuthology.orchestra.run.smithi061.stderr:11 Feb 06:07:49 ntpd[8414]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T06:07:49.045 INFO:teuthology.orchestra.run.smithi061.stderr:11 Feb 06:07:49 ntpd[8414]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T06:07:49.045 INFO:teuthology.orchestra.run.smithi061.stderr:11 Feb 06:07:49 ntpd[8414]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T06:07:50.044 INFO:teuthology.orchestra.run.smithi061.stderr:11 Feb 06:07:50 ntpd[8414]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T06:07:50.045 INFO:teuthology.orchestra.run.smithi061.stderr:11 Feb 06:07:50 ntpd[8414]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T06:07:50.045 INFO:teuthology.orchestra.run.smithi061.stderr:11 Feb 06:07:50 ntpd[8414]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T06:07:50.045 INFO:teuthology.orchestra.run.smithi061.stderr:11 Feb 06:07:50 ntpd[8414]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T06:07:50.045 INFO:teuthology.orchestra.run.smithi061.stderr:11 Feb 06:07:50 ntpd[8414]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T06:07:50.045 INFO:teuthology.orchestra.run.smithi061.stderr:11 Feb 06:07:50 ntpd[8414]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T06:07:50.045 INFO:teuthology.orchestra.run.smithi061.stderr:11 Feb 06:07:50 ntpd[8414]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T06:07:50.045 INFO:teuthology.orchestra.run.smithi061.stderr:11 Feb 06:07:50 ntpd[8414]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T06:07:51.044 INFO:teuthology.orchestra.run.smithi061.stderr:11 Feb 06:07:51 ntpd[8414]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T06:07:51.044 INFO:teuthology.orchestra.run.smithi061.stderr:11 Feb 06:07:51 ntpd[8414]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T06:07:51.044 INFO:teuthology.orchestra.run.smithi061.stderr:11 Feb 06:07:51 ntpd[8414]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T06:07:51.044 INFO:teuthology.orchestra.run.smithi061.stderr:11 Feb 06:07:51 ntpd[8414]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T06:07:51.044 INFO:teuthology.orchestra.run.smithi061.stderr:11 Feb 06:07:51 ntpd[8414]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T06:07:51.044 INFO:teuthology.orchestra.run.smithi061.stderr:11 Feb 06:07:51 ntpd[8414]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T06:07:51.045 INFO:teuthology.orchestra.run.smithi061.stderr:11 Feb 06:07:51 ntpd[8414]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T06:07:51.045 INFO:teuthology.orchestra.run.smithi061.stderr:11 Feb 06:07:51 ntpd[8414]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T06:07:52.044 INFO:teuthology.orchestra.run.smithi061.stderr:11 Feb 06:07:52 ntpd[8414]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T06:07:52.044 INFO:teuthology.orchestra.run.smithi061.stderr:11 Feb 06:07:52 ntpd[8414]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T06:07:52.044 INFO:teuthology.orchestra.run.smithi061.stderr:11 Feb 06:07:52 ntpd[8414]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T06:07:52.044 INFO:teuthology.orchestra.run.smithi061.stderr:11 Feb 06:07:52 ntpd[8414]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T06:07:52.044 INFO:teuthology.orchestra.run.smithi061.stderr:11 Feb 06:07:52 ntpd[8414]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T06:07:52.045 INFO:teuthology.orchestra.run.smithi061.stderr:11 Feb 06:07:52 ntpd[8414]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T06:07:52.045 INFO:teuthology.orchestra.run.smithi061.stderr:11 Feb 06:07:52 ntpd[8414]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T06:07:52.045 INFO:teuthology.orchestra.run.smithi061.stderr:11 Feb 06:07:52 ntpd[8414]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T06:07:53.045 INFO:teuthology.orchestra.run.smithi061.stderr:11 Feb 06:07:53 ntpd[8414]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T06:07:53.045 INFO:teuthology.orchestra.run.smithi061.stderr:11 Feb 06:07:53 ntpd[8414]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T06:07:53.045 INFO:teuthology.orchestra.run.smithi061.stderr:11 Feb 06:07:53 ntpd[8414]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T06:07:53.045 INFO:teuthology.orchestra.run.smithi061.stderr:11 Feb 06:07:53 ntpd[8414]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T06:07:53.046 INFO:teuthology.orchestra.run.smithi061.stderr:11 Feb 06:07:53 ntpd[8414]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T06:07:53.046 INFO:teuthology.orchestra.run.smithi061.stderr:11 Feb 06:07:53 ntpd[8414]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T06:07:53.046 INFO:teuthology.orchestra.run.smithi061.stderr:11 Feb 06:07:53 ntpd[8414]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T06:07:53.046 INFO:teuthology.orchestra.run.smithi061.stderr:11 Feb 06:07:53 ntpd[8414]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T06:07:54.044 INFO:teuthology.orchestra.run.smithi061.stderr:11 Feb 06:07:54 ntpd[8414]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T06:08:04.364 INFO:teuthology.orchestra.run.smithi061.stderr:11 Feb 06:07:54 ntpd[8414]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T06:08:04.364 INFO:teuthology.orchestra.run.smithi061.stderr:11 Feb 06:07:54 ntpd[8414]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T06:08:04.365 INFO:teuthology.orchestra.run.smithi061.stderr:11 Feb 06:07:54 ntpd[8414]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T06:08:04.365 INFO:teuthology.orchestra.run.smithi061.stderr:11 Feb 06:07:54 ntpd[8414]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T06:08:04.365 INFO:teuthology.orchestra.run.smithi061.stderr:11 Feb 06:07:54 ntpd[8414]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T06:08:04.365 INFO:teuthology.orchestra.run.smithi061.stderr:11 Feb 06:07:54 ntpd[8414]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T06:08:04.365 INFO:teuthology.orchestra.run.smithi061.stderr:11 Feb 06:07:54 ntpd[8414]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T06:08:04.366 INFO:teuthology.orchestra.run.smithi061.stderr:11 Feb 06:07:55 ntpd[8414]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T06:08:04.367 INFO:teuthology.orchestra.run.smithi061.stderr:11 Feb 06:07:55 ntpd[8414]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T06:08:04.367 INFO:teuthology.orchestra.run.smithi061.stderr:11 Feb 06:07:55 ntpd[8414]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T06:08:04.367 INFO:teuthology.orchestra.run.smithi061.stderr:11 Feb 06:07:55 ntpd[8414]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T06:08:04.367 INFO:teuthology.orchestra.run.smithi061.stderr:11 Feb 06:07:55 ntpd[8414]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2021-02-11T06:08:04.367 INFO:teuthology.orchestra.run.smithi061.stderr:11 Feb 06:07:55 ntpd[8414]: can't open /var/log/ntpstats/loopstats.20210211: Permission denied 2021-02-11T06:08:04.367 INFO:teuthology.orchestra.run.smithi061.stdout:11 Feb 06:07:55 ntpd[8414]: ntpd: time slew -0.002335 s 2021-02-11T06:08:04.367 INFO:teuthology.orchestra.run.smithi061.stdout:ntpd: time slew -0.002335s 2021-02-11T06:08:04.367 INFO:teuthology.orchestra.run.smithi061.stdout: remote refid st t when poll reach delay offset jitter 2021-02-11T06:08:04.367 INFO:teuthology.orchestra.run.smithi061.stdout:============================================================================== 2021-02-11T06:08:04.367 INFO:teuthology.orchestra.run.smithi061.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-02-11T06:08:04.367 INFO:teuthology.orchestra.run.smithi061.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-02-11T06:08:04.367 INFO:teuthology.orchestra.run.smithi061.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-02-11T06:08:04.367 INFO:teuthology.orchestra.run.smithi061.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-02-11T06:08:04.368 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-02-11T06:08:04.368 DEBUG:teuthology.orchestra.run.smithi172:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-02-11T06:08:04.424 INFO:teuthology.orchestra.run.smithi172.stdout:11 Feb 06:08:04 ntpd[8431]: ntpd 4.2.8p10@1.3728-o (1): Starting 2021-02-11T06:08:04.424 INFO:teuthology.orchestra.run.smithi172.stdout:11 Feb 06:08:04 ntpd[8431]: Command line: ntpd -gq 2021-02-11T06:08:04.425 INFO:teuthology.orchestra.run.smithi172.stdout:11 Feb 06:08:04 ntpd[8431]: proto: precision = 0.120 usec (-23) 2021-02-11T06:08:04.425 INFO:teuthology.orchestra.run.smithi172.stdout:11 Feb 06:08:04 ntpd[8431]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2021-02-11T06:08:04.425 INFO:teuthology.orchestra.run.smithi172.stdout:11 Feb 06:08:04 ntpd[8431]: restrict ::: KOD does nothing without LIMITED. 2021-02-11T06:08:04.425 INFO:teuthology.orchestra.run.smithi172.stdout:11 Feb 06:08:04 ntpd[8431]: Listen and drop on 0 v6wildcard [::]:123 2021-02-11T06:08:04.425 INFO:teuthology.orchestra.run.smithi172.stdout:11 Feb 06:08:04 ntpd[8431]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2021-02-11T06:08:04.426 INFO:teuthology.orchestra.run.smithi172.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2021-02-11T06:08:04.426 INFO:teuthology.orchestra.run.smithi172.stderr:restrict ::: KOD does nothing without LIMITED. 2021-02-11T06:08:04.426 INFO:teuthology.orchestra.run.smithi172.stdout:11 Feb 06:08:04 ntpd[8431]: Listen normally on 2 lo 127.0.0.1:123 2021-02-11T06:08:04.426 INFO:teuthology.orchestra.run.smithi172.stdout:11 Feb 06:08:04 ntpd[8431]: Listen normally on 3 enp3s0f1 172.21.15.172:123 2021-02-11T06:08:04.427 INFO:teuthology.orchestra.run.smithi172.stdout:11 Feb 06:08:04 ntpd[8431]: Listen normally on 4 lo [::1]:123 2021-02-11T06:08:04.427 INFO:teuthology.orchestra.run.smithi172.stdout:11 Feb 06:08:04 ntpd[8431]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7f91%5]:123 2021-02-11T06:08:04.427 INFO:teuthology.orchestra.run.smithi172.stdout:11 Feb 06:08:04 ntpd[8431]: Listening on routing socket on fd #22 for interface updates 2021-02-11T06:08:05.425 INFO:teuthology.orchestra.run.smithi172.stderr:11 Feb 06:08:05 ntpd[8431]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T06:08:06.632 INFO:teuthology.orchestra.run.smithi172.stderr:11 Feb 06:08:05 ntpd[8431]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T06:08:06.632 INFO:teuthology.orchestra.run.smithi172.stderr:11 Feb 06:08:05 ntpd[8431]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T06:08:06.632 INFO:teuthology.orchestra.run.smithi172.stderr:11 Feb 06:08:05 ntpd[8431]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T06:08:06.632 INFO:teuthology.orchestra.run.smithi172.stderr:11 Feb 06:08:05 ntpd[8431]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T06:08:06.633 INFO:teuthology.orchestra.run.smithi172.stderr:11 Feb 06:08:05 ntpd[8431]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T06:08:06.633 INFO:teuthology.orchestra.run.smithi172.stderr:11 Feb 06:08:05 ntpd[8431]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T06:08:06.633 INFO:teuthology.orchestra.run.smithi172.stderr:11 Feb 06:08:05 ntpd[8431]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T06:08:07.425 INFO:teuthology.orchestra.run.smithi172.stderr:11 Feb 06:08:07 ntpd[8431]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T06:08:08.607 INFO:teuthology.orchestra.run.smithi172.stderr:11 Feb 06:08:07 ntpd[8431]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T06:08:08.608 INFO:teuthology.orchestra.run.smithi172.stderr:11 Feb 06:08:07 ntpd[8431]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T06:08:08.608 INFO:teuthology.orchestra.run.smithi172.stderr:11 Feb 06:08:07 ntpd[8431]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T06:08:08.608 INFO:teuthology.orchestra.run.smithi172.stderr:11 Feb 06:08:07 ntpd[8431]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T06:08:08.608 INFO:teuthology.orchestra.run.smithi172.stderr:11 Feb 06:08:07 ntpd[8431]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T06:08:08.608 INFO:teuthology.orchestra.run.smithi172.stderr:11 Feb 06:08:07 ntpd[8431]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T06:08:08.608 INFO:teuthology.orchestra.run.smithi172.stderr:11 Feb 06:08:07 ntpd[8431]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T06:08:08.609 INFO:teuthology.orchestra.run.smithi172.stderr:11 Feb 06:08:07 ntpd[8431]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T06:08:08.609 INFO:teuthology.orchestra.run.smithi172.stderr:11 Feb 06:08:07 ntpd[8431]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T06:08:08.609 INFO:teuthology.orchestra.run.smithi172.stderr:11 Feb 06:08:07 ntpd[8431]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T06:08:08.609 INFO:teuthology.orchestra.run.smithi172.stderr:11 Feb 06:08:07 ntpd[8431]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T06:08:08.609 INFO:teuthology.orchestra.run.smithi172.stderr:11 Feb 06:08:07 ntpd[8431]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T06:08:08.609 INFO:teuthology.orchestra.run.smithi172.stderr:11 Feb 06:08:07 ntpd[8431]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T06:08:08.610 INFO:teuthology.orchestra.run.smithi172.stderr:11 Feb 06:08:07 ntpd[8431]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T06:08:08.610 INFO:teuthology.orchestra.run.smithi172.stderr:11 Feb 06:08:07 ntpd[8431]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T06:08:09.425 INFO:teuthology.orchestra.run.smithi172.stderr:11 Feb 06:08:09 ntpd[8431]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T06:08:09.425 INFO:teuthology.orchestra.run.smithi172.stderr:11 Feb 06:08:09 ntpd[8431]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T06:08:09.426 INFO:teuthology.orchestra.run.smithi172.stderr:11 Feb 06:08:09 ntpd[8431]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T06:08:09.426 INFO:teuthology.orchestra.run.smithi172.stderr:11 Feb 06:08:09 ntpd[8431]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T06:08:09.426 INFO:teuthology.orchestra.run.smithi172.stderr:11 Feb 06:08:09 ntpd[8431]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T06:08:09.426 INFO:teuthology.orchestra.run.smithi172.stderr:11 Feb 06:08:09 ntpd[8431]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T06:08:09.426 INFO:teuthology.orchestra.run.smithi172.stderr:11 Feb 06:08:09 ntpd[8431]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T06:08:09.426 INFO:teuthology.orchestra.run.smithi172.stderr:11 Feb 06:08:09 ntpd[8431]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T06:08:09.426 INFO:teuthology.orchestra.run.smithi172.stderr:11 Feb 06:08:09 ntpd[8431]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T06:08:09.426 INFO:teuthology.orchestra.run.smithi172.stderr:11 Feb 06:08:09 ntpd[8431]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T06:08:09.426 INFO:teuthology.orchestra.run.smithi172.stderr:11 Feb 06:08:09 ntpd[8431]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T06:08:09.426 INFO:teuthology.orchestra.run.smithi172.stderr:11 Feb 06:08:09 ntpd[8431]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T06:08:09.426 INFO:teuthology.orchestra.run.smithi172.stderr:11 Feb 06:08:09 ntpd[8431]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T06:08:09.426 INFO:teuthology.orchestra.run.smithi172.stderr:11 Feb 06:08:09 ntpd[8431]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T06:08:09.426 INFO:teuthology.orchestra.run.smithi172.stderr:11 Feb 06:08:09 ntpd[8431]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T06:08:09.427 INFO:teuthology.orchestra.run.smithi172.stderr:11 Feb 06:08:09 ntpd[8431]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T06:08:11.425 INFO:teuthology.orchestra.run.smithi172.stderr:11 Feb 06:08:11 ntpd[8431]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T06:08:11.425 INFO:teuthology.orchestra.run.smithi172.stderr:11 Feb 06:08:11 ntpd[8431]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T06:08:11.426 INFO:teuthology.orchestra.run.smithi172.stderr:11 Feb 06:08:11 ntpd[8431]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T06:08:11.426 INFO:teuthology.orchestra.run.smithi172.stderr:11 Feb 06:08:11 ntpd[8431]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T06:08:11.426 INFO:teuthology.orchestra.run.smithi172.stderr:11 Feb 06:08:11 ntpd[8431]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2021-02-11T06:08:11.426 INFO:teuthology.orchestra.run.smithi172.stderr:11 Feb 06:08:11 ntpd[8431]: can't open /var/log/ntpstats/loopstats.20210211: Permission denied 2021-02-11T06:08:11.426 INFO:teuthology.orchestra.run.smithi172.stdout:11 Feb 06:08:11 ntpd[8431]: ntpd: time slew -0.002181 s 2021-02-11T06:08:11.427 INFO:teuthology.orchestra.run.smithi172.stdout:ntpd: time slew -0.002181s 2021-02-11T06:08:11.489 INFO:teuthology.orchestra.run.smithi172.stdout: remote refid st t when poll reach delay offset jitter 2021-02-11T06:08:11.489 INFO:teuthology.orchestra.run.smithi172.stdout:============================================================================== 2021-02-11T06:08:11.489 INFO:teuthology.orchestra.run.smithi172.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-02-11T06:08:11.490 INFO:teuthology.orchestra.run.smithi172.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-02-11T06:08:11.490 INFO:teuthology.orchestra.run.smithi172.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-02-11T06:08:11.490 INFO:teuthology.orchestra.run.smithi172.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-02-11T06:08:11.490 INFO:teuthology.run_tasks:Running task install... 2021-02-11T06:08:11.496 DEBUG:teuthology.task.install:project ceph 2021-02-11T06:08:11.496 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': '84a8025e5a892a6879f75e17e03444bf443eae4b'}} 2021-02-11T06:08:11.496 DEBUG:teuthology.task.install:config {'sha1': '84a8025e5a892a6879f75e17e03444bf443eae4b'} 2021-02-11T06:08:11.496 INFO:teuthology.task.install:Using flavor: basic 2021-02-11T06:08:11.513 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2021-02-11T06:08:11.513 INFO:teuthology.task.install:extra packages: [] 2021-02-11T06:08:11.513 DEBUG:teuthology.orchestra.run.smithi061:> sudo lsb_release -is 2021-02-11T06:08:11.566 INFO:teuthology.orchestra.run.smithi061.stdout:Ubuntu 2021-02-11T06:08:11.571 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-02-11T06:08:11.572 DEBUG:teuthology.orchestra.run.smithi172:> sudo lsb_release -is 2021-02-11T06:08:11.572 DEBUG:teuthology.orchestra.run.smithi061:> sudo apt-key list | grep Ceph 2021-02-11T06:08:11.626 INFO:teuthology.orchestra.run.smithi172.stdout:Ubuntu 2021-02-11T06:08:11.631 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-02-11T06:08:11.631 DEBUG:teuthology.orchestra.run.smithi172:> sudo apt-key list | grep Ceph 2021-02-11T06:08:11.670 INFO:teuthology.orchestra.run.smithi061.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2021-02-11T06:08:11.720 INFO:teuthology.orchestra.run.smithi172.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2021-02-11T06:08:11.726 INFO:teuthology.orchestra.run.smithi061.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2021-02-11T06:08:11.726 INFO:teuthology.orchestra.run.smithi061.stdout:uid [ unknown] Ceph.com (release key) 2021-02-11T06:08:11.727 DEBUG:teuthology.orchestra.run.smithi061:> sudo lsb_release -is 2021-02-11T06:08:11.772 INFO:teuthology.orchestra.run.smithi172.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2021-02-11T06:08:11.772 INFO:teuthology.orchestra.run.smithi172.stdout:uid [ unknown] Ceph.com (release key) 2021-02-11T06:08:11.773 DEBUG:teuthology.orchestra.run.smithi172:> sudo lsb_release -is 2021-02-11T06:08:11.777 INFO:teuthology.orchestra.run.smithi061.stdout:Ubuntu 2021-02-11T06:08:11.783 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-02-11T06:08:11.783 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2021-02-11T06:08:11.783 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=84a8025e5a892a6879f75e17e03444bf443eae4b 2021-02-11T06:08:11.823 INFO:teuthology.orchestra.run.smithi172.stdout:Ubuntu 2021-02-11T06:08:11.829 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-02-11T06:08:11.829 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2021-02-11T06:08:11.829 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=84a8025e5a892a6879f75e17e03444bf443eae4b 2021-02-11T06:08:11.958 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/wip-easter-bunny/84a8025e5a892a6879f75e17e03444bf443eae4b/ubuntu/bionic/flavors/default/ 2021-02-11T06:08:11.958 INFO:teuthology.task.install.deb:Package version is 17.0.0-711-g84a8025e-1bionic 2021-02-11T06:08:11.997 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/wip-easter-bunny/84a8025e5a892a6879f75e17e03444bf443eae4b/ubuntu/bionic/flavors/default/ 2021-02-11T06:08:11.997 INFO:teuthology.task.install.deb:Package version is 17.0.0-711-g84a8025e-1bionic 2021-02-11T06:08:12.086 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2021-02-11T06:08:12.086 DEBUG:teuthology.orchestra.run.smithi061:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2021-02-11T06:08:12.102 DEBUG:teuthology.orchestra.run.smithi061:> sudo apt-get update 2021-02-11T06:08:12.116 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2021-02-11T06:08:12.117 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2021-02-11T06:08:12.130 DEBUG:teuthology.orchestra.run.smithi172:> sudo apt-get update 2021-02-11T06:08:12.317 INFO:teuthology.orchestra.run.smithi061.stdout:Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease 2021-02-11T06:08:12.369 INFO:teuthology.orchestra.run.smithi061.stdout:Ign:2 https://1.chacra.ceph.com/r/ceph/wip-easter-bunny/84a8025e5a892a6879f75e17e03444bf443eae4b/ubuntu/bionic/flavors/default bionic InRelease 2021-02-11T06:08:12.373 INFO:teuthology.orchestra.run.smithi172.stdout:Ign:1 https://1.chacra.ceph.com/r/ceph/wip-easter-bunny/84a8025e5a892a6879f75e17e03444bf443eae4b/ubuntu/bionic/flavors/default bionic InRelease 2021-02-11T06:08:12.393 INFO:teuthology.orchestra.run.smithi061.stdout:Get:3 https://1.chacra.ceph.com/r/ceph/wip-easter-bunny/84a8025e5a892a6879f75e17e03444bf443eae4b/ubuntu/bionic/flavors/default bionic Release [7,681 B] 2021-02-11T06:08:12.396 INFO:teuthology.orchestra.run.smithi172.stdout:Get:2 https://1.chacra.ceph.com/r/ceph/wip-easter-bunny/84a8025e5a892a6879f75e17e03444bf443eae4b/ubuntu/bionic/flavors/default bionic Release [7,681 B] 2021-02-11T06:08:12.418 INFO:teuthology.orchestra.run.smithi061.stdout:Ign:4 https://1.chacra.ceph.com/r/ceph/wip-easter-bunny/84a8025e5a892a6879f75e17e03444bf443eae4b/ubuntu/bionic/flavors/default bionic Release.gpg 2021-02-11T06:08:12.421 INFO:teuthology.orchestra.run.smithi061.stdout:Hit:5 http://archive.ubuntu.com/ubuntu bionic InRelease 2021-02-11T06:08:12.432 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:3 http://archive.ubuntu.com/ubuntu bionic InRelease 2021-02-11T06:08:12.432 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:4 http://security.ubuntu.com/ubuntu bionic-security InRelease 2021-02-11T06:08:12.433 INFO:teuthology.orchestra.run.smithi172.stdout:Ign:5 https://1.chacra.ceph.com/r/ceph/wip-easter-bunny/84a8025e5a892a6879f75e17e03444bf443eae4b/ubuntu/bionic/flavors/default bionic Release.gpg 2021-02-11T06:08:12.442 INFO:teuthology.orchestra.run.smithi061.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/wip-easter-bunny/84a8025e5a892a6879f75e17e03444bf443eae4b/ubuntu/bionic/flavors/default bionic/main amd64 Packages [16.6 kB] 2021-02-11T06:08:12.457 INFO:teuthology.orchestra.run.smithi172.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/wip-easter-bunny/84a8025e5a892a6879f75e17e03444bf443eae4b/ubuntu/bionic/flavors/default bionic/main i386 Packages [3,110 B] 2021-02-11T06:08:12.483 INFO:teuthology.orchestra.run.smithi061.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/wip-easter-bunny/84a8025e5a892a6879f75e17e03444bf443eae4b/ubuntu/bionic/flavors/default bionic/main i386 Packages [3,110 B] 2021-02-11T06:08:12.493 INFO:teuthology.orchestra.run.smithi172.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/wip-easter-bunny/84a8025e5a892a6879f75e17e03444bf443eae4b/ubuntu/bionic/flavors/default bionic/main amd64 Packages [16.6 kB] 2021-02-11T06:08:12.497 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:8 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2021-02-11T06:08:12.501 INFO:teuthology.orchestra.run.smithi061.stdout:Hit:8 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2021-02-11T06:08:12.576 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:9 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2021-02-11T06:08:12.580 INFO:teuthology.orchestra.run.smithi061.stdout:Hit:9 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2021-02-11T06:08:12.940 INFO:teuthology.orchestra.run.smithi061.stdout:Fetched 27.3 kB in 1s (43.6 kB/s) 2021-02-11T06:08:36.725 INFO:teuthology.orchestra.run.smithi172.stdout:Fetched 27.3 kB in 1s (39.0 kB/s) 2021-02-11T06:08:36.725 INFO:teuthology.orchestra.run.smithi172.stdout:Reading package lists... 2021-02-11T06:08:36.725 INFO:teuthology.orchestra.run.smithi061.stdout:Reading package lists... 2021-02-11T06:08:36.726 DEBUG:teuthology.orchestra.run.smithi061:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.0.0-711-g84a8025e-1bionic cephadm=17.0.0-711-g84a8025e-1bionic ceph-mds=17.0.0-711-g84a8025e-1bionic ceph-mgr=17.0.0-711-g84a8025e-1bionic ceph-common=17.0.0-711-g84a8025e-1bionic ceph-fuse=17.0.0-711-g84a8025e-1bionic ceph-test=17.0.0-711-g84a8025e-1bionic radosgw=17.0.0-711-g84a8025e-1bionic python3-rados=17.0.0-711-g84a8025e-1bionic python3-rgw=17.0.0-711-g84a8025e-1bionic python3-cephfs=17.0.0-711-g84a8025e-1bionic python3-rbd=17.0.0-711-g84a8025e-1bionic libcephfs2=17.0.0-711-g84a8025e-1bionic libcephfs-dev=17.0.0-711-g84a8025e-1bionic librados2=17.0.0-711-g84a8025e-1bionic librbd1=17.0.0-711-g84a8025e-1bionic rbd-fuse=17.0.0-711-g84a8025e-1bionic 2021-02-11T06:08:36.726 DEBUG:teuthology.orchestra.run.smithi172:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.0.0-711-g84a8025e-1bionic cephadm=17.0.0-711-g84a8025e-1bionic ceph-mds=17.0.0-711-g84a8025e-1bionic ceph-mgr=17.0.0-711-g84a8025e-1bionic ceph-common=17.0.0-711-g84a8025e-1bionic ceph-fuse=17.0.0-711-g84a8025e-1bionic ceph-test=17.0.0-711-g84a8025e-1bionic radosgw=17.0.0-711-g84a8025e-1bionic python3-rados=17.0.0-711-g84a8025e-1bionic python3-rgw=17.0.0-711-g84a8025e-1bionic python3-cephfs=17.0.0-711-g84a8025e-1bionic python3-rbd=17.0.0-711-g84a8025e-1bionic libcephfs2=17.0.0-711-g84a8025e-1bionic libcephfs-dev=17.0.0-711-g84a8025e-1bionic librados2=17.0.0-711-g84a8025e-1bionic librbd1=17.0.0-711-g84a8025e-1bionic rbd-fuse=17.0.0-711-g84a8025e-1bionic 2021-02-11T06:08:36.813 INFO:teuthology.orchestra.run.smithi172.stdout:Reading package lists... 2021-02-11T06:08:36.814 INFO:teuthology.orchestra.run.smithi061.stdout:Reading package lists... 2021-02-11T06:08:37.058 INFO:teuthology.orchestra.run.smithi172.stdout:Building dependency tree... 2021-02-11T06:08:37.058 INFO:teuthology.orchestra.run.smithi172.stdout:Reading state information... 2021-02-11T06:08:37.081 INFO:teuthology.orchestra.run.smithi061.stdout:Building dependency tree... 2021-02-11T06:08:37.081 INFO:teuthology.orchestra.run.smithi061.stdout:Reading state information... 2021-02-11T06:08:37.226 INFO:teuthology.orchestra.run.smithi172.stdout:The following additional packages will be installed: 2021-02-11T06:08:37.226 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2021-02-11T06:08:37.226 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd jq 2021-02-11T06:08:37.226 INFO:teuthology.orchestra.run.smithi172.stdout: libjaeger libjq1 libleveldb1v5 liblttng-ust0 liboath0 libonig4 librabbitmq4 2021-02-11T06:08:37.227 INFO:teuthology.orchestra.run.smithi172.stdout: libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libxslt1.1 nvme-cli 2021-02-11T06:08:37.227 INFO:teuthology.orchestra.run.smithi172.stdout: python-attr python-funcsigs python-pastedeploy-tpl python-pluggy python-py 2021-02-11T06:08:37.227 INFO:teuthology.orchestra.run.smithi172.stdout: python-pytest python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2021-02-11T06:08:37.227 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ceph-argparse python3-ceph-common python3-cffi-backend 2021-02-11T06:08:37.227 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cherrypy3 python3-cryptography python3-cycler python3-dateutil 2021-02-11T06:08:37.227 INFO:teuthology.orchestra.run.smithi172.stdout: python3-decorator python3-fasteners python3-html5lib python3-httplib2 2021-02-11T06:08:37.227 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jinja2 python3-joblib python3-json-pointer python3-jsonpatch 2021-02-11T06:08:37.227 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2021-02-11T06:08:37.227 INFO:teuthology.orchestra.run.smithi172.stdout: python3-markupsafe python3-matplotlib python3-monotonic python3-numpy 2021-02-11T06:08:37.228 INFO:teuthology.orchestra.run.smithi172.stdout: python3-oauth2client python3-olefile python3-openssl python3-paste 2021-02-11T06:08:37.228 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pastedeploy python3-pastescript python3-pecan python3-pil 2021-02-11T06:08:37.228 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pluggy python3-prettytable python3-py python3-pyasn1 2021-02-11T06:08:37.228 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyasn1-modules python3-pyinotify python3-pyparsing python3-pytest 2021-02-11T06:08:37.228 INFO:teuthology.orchestra.run.smithi172.stdout: python3-repoze.lru python3-routes python3-rsa python3-scipy 2021-02-11T06:08:37.228 INFO:teuthology.orchestra.run.smithi172.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2021-02-11T06:08:37.228 INFO:teuthology.orchestra.run.smithi172.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2021-02-11T06:08:37.228 INFO:teuthology.orchestra.run.smithi172.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2021-02-11T06:08:37.228 INFO:teuthology.orchestra.run.smithi172.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2021-02-11T06:08:37.228 INFO:teuthology.orchestra.run.smithi172.stdout: python3-webtest python3-werkzeug socat xmlstarlet 2021-02-11T06:08:37.229 INFO:teuthology.orchestra.run.smithi172.stdout:Suggested packages: 2021-02-11T06:08:37.229 INFO:teuthology.orchestra.run.smithi172.stdout: python3-influxdb python-attr-doc python-funcsigs-doc subversion 2021-02-11T06:08:37.229 INFO:teuthology.orchestra.run.smithi172.stdout: python-pytest-xdist python-mock python-cryptography-doc 2021-02-11T06:08:37.229 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cryptography-vectors python-cycler-doc python3-genshi 2021-02-11T06:08:37.229 INFO:teuthology.orchestra.run.smithi172.stdout: python-jinja2-doc python3-crypto python3-lxml-dbg python-lxml-doc 2021-02-11T06:08:37.229 INFO:teuthology.orchestra.run.smithi172.stdout: python3-beaker python-mako-doc dvipng ffmpeg gir1.2-gtk-3.0 ghostscript 2021-02-11T06:08:37.229 INFO:teuthology.orchestra.run.smithi172.stdout: inkscape ipython3 python-matplotlib-doc python3-cairocffi python3-gi-cairo 2021-02-11T06:08:37.229 INFO:teuthology.orchestra.run.smithi172.stdout: python3-gobject python3-pyqt4 python3-sip python3-tornado 2021-02-11T06:08:37.229 INFO:teuthology.orchestra.run.smithi172.stdout: texlive-extra-utils texlive-latex-extra ttf-staypuft python-numpy-doc 2021-02-11T06:08:37.229 INFO:teuthology.orchestra.run.smithi172.stdout: python3-dev python3-numpy-dbg python-openssl-doc python3-openssl-dbg 2021-02-11T06:08:37.229 INFO:teuthology.orchestra.run.smithi172.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2021-02-11T06:08:37.229 INFO:teuthology.orchestra.run.smithi172.stdout: python-pecan-doc python-pil-doc python3-pil-dbg python-pyinotify-doc 2021-02-11T06:08:37.230 INFO:teuthology.orchestra.run.smithi172.stdout: python-pyparsing-doc python-scipy-doc python-setuptools-doc python3-dap 2021-02-11T06:08:37.230 INFO:teuthology.orchestra.run.smithi172.stdout: python-sklearn-doc python-sqlalchemy-doc python3-psycopg2 python3-pymysql 2021-02-11T06:08:37.230 INFO:teuthology.orchestra.run.smithi172.stdout: python3-fdb tix python3-tk-dbg python-waitress-doc python-webob-doc 2021-02-11T06:08:37.230 INFO:teuthology.orchestra.run.smithi172.stdout: python-webtest-doc python3-termcolor python3-watchdog python-werkzeug-doc 2021-02-11T06:08:37.262 INFO:teuthology.orchestra.run.smithi061.stdout:The following additional packages will be installed: 2021-02-11T06:08:37.262 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2021-02-11T06:08:37.262 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd jq 2021-02-11T06:08:37.262 INFO:teuthology.orchestra.run.smithi061.stdout: libjaeger libjq1 libleveldb1v5 liblttng-ust0 liboath0 libonig4 librabbitmq4 2021-02-11T06:08:37.263 INFO:teuthology.orchestra.run.smithi061.stdout: libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libxslt1.1 nvme-cli 2021-02-11T06:08:37.263 INFO:teuthology.orchestra.run.smithi061.stdout: python-attr python-funcsigs python-pastedeploy-tpl python-pluggy python-py 2021-02-11T06:08:37.263 INFO:teuthology.orchestra.run.smithi061.stdout: python-pytest python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2021-02-11T06:08:37.263 INFO:teuthology.orchestra.run.smithi061.stdout: python3-ceph-argparse python3-ceph-common python3-cffi-backend 2021-02-11T06:08:37.263 INFO:teuthology.orchestra.run.smithi061.stdout: python3-cherrypy3 python3-cryptography python3-cycler python3-dateutil 2021-02-11T06:08:37.264 INFO:teuthology.orchestra.run.smithi061.stdout: python3-decorator python3-fasteners python3-html5lib python3-httplib2 2021-02-11T06:08:37.264 INFO:teuthology.orchestra.run.smithi061.stdout: python3-jinja2 python3-joblib python3-json-pointer python3-jsonpatch 2021-02-11T06:08:37.264 INFO:teuthology.orchestra.run.smithi061.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2021-02-11T06:08:37.264 INFO:teuthology.orchestra.run.smithi061.stdout: python3-markupsafe python3-matplotlib python3-monotonic python3-numpy 2021-02-11T06:08:37.264 INFO:teuthology.orchestra.run.smithi061.stdout: python3-oauth2client python3-olefile python3-openssl python3-paste 2021-02-11T06:08:37.264 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pastedeploy python3-pastescript python3-pecan python3-pil 2021-02-11T06:08:37.264 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pluggy python3-prettytable python3-py python3-pyasn1 2021-02-11T06:08:37.264 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pyasn1-modules python3-pyinotify python3-pyparsing python3-pytest 2021-02-11T06:08:37.264 INFO:teuthology.orchestra.run.smithi061.stdout: python3-repoze.lru python3-routes python3-rsa python3-scipy 2021-02-11T06:08:37.264 INFO:teuthology.orchestra.run.smithi061.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2021-02-11T06:08:37.264 INFO:teuthology.orchestra.run.smithi061.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2021-02-11T06:08:37.265 INFO:teuthology.orchestra.run.smithi061.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2021-02-11T06:08:37.265 INFO:teuthology.orchestra.run.smithi061.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2021-02-11T06:08:37.265 INFO:teuthology.orchestra.run.smithi061.stdout: python3-webtest python3-werkzeug socat xmlstarlet 2021-02-11T06:08:37.265 INFO:teuthology.orchestra.run.smithi061.stdout:Suggested packages: 2021-02-11T06:08:37.265 INFO:teuthology.orchestra.run.smithi061.stdout: python3-influxdb python-attr-doc python-funcsigs-doc subversion 2021-02-11T06:08:37.265 INFO:teuthology.orchestra.run.smithi061.stdout: python-pytest-xdist python-mock python-cryptography-doc 2021-02-11T06:08:37.265 INFO:teuthology.orchestra.run.smithi061.stdout: python3-cryptography-vectors python-cycler-doc python3-genshi 2021-02-11T06:08:37.265 INFO:teuthology.orchestra.run.smithi061.stdout: python-jinja2-doc python3-crypto python3-lxml-dbg python-lxml-doc 2021-02-11T06:08:37.266 INFO:teuthology.orchestra.run.smithi061.stdout: python3-beaker python-mako-doc dvipng ffmpeg gir1.2-gtk-3.0 ghostscript 2021-02-11T06:08:37.266 INFO:teuthology.orchestra.run.smithi061.stdout: inkscape ipython3 python-matplotlib-doc python3-cairocffi python3-gi-cairo 2021-02-11T06:08:37.266 INFO:teuthology.orchestra.run.smithi061.stdout: python3-gobject python3-pyqt4 python3-sip python3-tornado 2021-02-11T06:08:37.266 INFO:teuthology.orchestra.run.smithi061.stdout: texlive-extra-utils texlive-latex-extra ttf-staypuft python-numpy-doc 2021-02-11T06:08:37.266 INFO:teuthology.orchestra.run.smithi061.stdout: python3-dev python3-numpy-dbg python-openssl-doc python3-openssl-dbg 2021-02-11T06:08:37.266 INFO:teuthology.orchestra.run.smithi061.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2021-02-11T06:08:37.266 INFO:teuthology.orchestra.run.smithi061.stdout: python-pecan-doc python-pil-doc python3-pil-dbg python-pyinotify-doc 2021-02-11T06:08:37.266 INFO:teuthology.orchestra.run.smithi061.stdout: python-pyparsing-doc python-scipy-doc python-setuptools-doc python3-dap 2021-02-11T06:08:37.266 INFO:teuthology.orchestra.run.smithi061.stdout: python-sklearn-doc python-sqlalchemy-doc python3-psycopg2 python3-pymysql 2021-02-11T06:08:37.266 INFO:teuthology.orchestra.run.smithi061.stdout: python3-fdb tix python3-tk-dbg python-waitress-doc python-webob-doc 2021-02-11T06:08:37.266 INFO:teuthology.orchestra.run.smithi061.stdout: python-webtest-doc python3-termcolor python3-watchdog python-werkzeug-doc 2021-02-11T06:08:37.331 INFO:teuthology.orchestra.run.smithi172.stdout:The following NEW packages will be installed: 2021-02-11T06:08:37.332 INFO:teuthology.orchestra.run.smithi172.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2021-02-11T06:08:37.332 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2021-02-11T06:08:37.332 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd ceph-test cephadm jq 2021-02-11T06:08:37.332 INFO:teuthology.orchestra.run.smithi172.stdout: libcephfs-dev libcephfs2 libjaeger libjq1 libleveldb1v5 liblttng-ust0 2021-02-11T06:08:37.332 INFO:teuthology.orchestra.run.smithi172.stdout: liboath0 libonig4 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2021-02-11T06:08:37.333 INFO:teuthology.orchestra.run.smithi172.stdout: libsnappy1v5 libxslt1.1 nvme-cli python-attr python-funcsigs 2021-02-11T06:08:37.333 INFO:teuthology.orchestra.run.smithi172.stdout: python-pastedeploy-tpl python-pluggy python-py python-pytest 2021-02-11T06:08:37.333 INFO:teuthology.orchestra.run.smithi172.stdout: python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2021-02-11T06:08:37.334 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs 2021-02-11T06:08:37.334 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cffi-backend python3-cherrypy3 python3-cryptography python3-cycler 2021-02-11T06:08:37.334 INFO:teuthology.orchestra.run.smithi172.stdout: python3-dateutil python3-decorator python3-fasteners python3-html5lib 2021-02-11T06:08:37.334 INFO:teuthology.orchestra.run.smithi172.stdout: python3-httplib2 python3-jinja2 python3-joblib python3-json-pointer 2021-02-11T06:08:37.334 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jsonpatch python3-jwt python3-kubernetes python3-logutils 2021-02-11T06:08:37.334 INFO:teuthology.orchestra.run.smithi172.stdout: python3-lxml python3-mako python3-markupsafe python3-matplotlib 2021-02-11T06:08:37.334 INFO:teuthology.orchestra.run.smithi172.stdout: python3-monotonic python3-numpy python3-oauth2client python3-olefile 2021-02-11T06:08:37.334 INFO:teuthology.orchestra.run.smithi172.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2021-02-11T06:08:37.334 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py 2021-02-11T06:08:37.334 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyparsing 2021-02-11T06:08:37.334 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-rgw 2021-02-11T06:08:37.334 INFO:teuthology.orchestra.run.smithi172.stdout: python3-routes python3-rsa python3-scipy python3-setuptools 2021-02-11T06:08:37.334 INFO:teuthology.orchestra.run.smithi172.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2021-02-11T06:08:37.335 INFO:teuthology.orchestra.run.smithi172.stdout: python3-sklearn python3-sklearn-lib python3-sqlalchemy python3-tempita 2021-02-11T06:08:37.335 INFO:teuthology.orchestra.run.smithi172.stdout: python3-tk python3-tz python3-uritemplate python3-waitress 2021-02-11T06:08:37.335 INFO:teuthology.orchestra.run.smithi172.stdout: python3-webencodings python3-webob python3-websocket python3-webtest 2021-02-11T06:08:37.335 INFO:teuthology.orchestra.run.smithi172.stdout: python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2021-02-11T06:08:37.335 INFO:teuthology.orchestra.run.smithi172.stdout:The following packages will be upgraded: 2021-02-11T06:08:37.335 INFO:teuthology.orchestra.run.smithi172.stdout: librados2 librbd1 2021-02-11T06:08:37.369 INFO:teuthology.orchestra.run.smithi061.stdout:The following NEW packages will be installed: 2021-02-11T06:08:37.370 INFO:teuthology.orchestra.run.smithi061.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2021-02-11T06:08:37.370 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2021-02-11T06:08:37.370 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd ceph-test cephadm jq 2021-02-11T06:08:37.370 INFO:teuthology.orchestra.run.smithi061.stdout: libcephfs-dev libcephfs2 libjaeger libjq1 libleveldb1v5 liblttng-ust0 2021-02-11T06:08:37.370 INFO:teuthology.orchestra.run.smithi061.stdout: liboath0 libonig4 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2021-02-11T06:08:37.371 INFO:teuthology.orchestra.run.smithi061.stdout: libsnappy1v5 libxslt1.1 nvme-cli python-attr python-funcsigs 2021-02-11T06:08:37.371 INFO:teuthology.orchestra.run.smithi061.stdout: python-pastedeploy-tpl python-pluggy python-py python-pytest 2021-02-11T06:08:37.371 INFO:teuthology.orchestra.run.smithi061.stdout: python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2021-02-11T06:08:37.372 INFO:teuthology.orchestra.run.smithi061.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs 2021-02-11T06:08:37.372 INFO:teuthology.orchestra.run.smithi061.stdout: python3-cffi-backend python3-cherrypy3 python3-cryptography python3-cycler 2021-02-11T06:08:37.372 INFO:teuthology.orchestra.run.smithi061.stdout: python3-dateutil python3-decorator python3-fasteners python3-html5lib 2021-02-11T06:08:37.372 INFO:teuthology.orchestra.run.smithi061.stdout: python3-httplib2 python3-jinja2 python3-joblib python3-json-pointer 2021-02-11T06:08:37.372 INFO:teuthology.orchestra.run.smithi061.stdout: python3-jsonpatch python3-jwt python3-kubernetes python3-logutils 2021-02-11T06:08:37.372 INFO:teuthology.orchestra.run.smithi061.stdout: python3-lxml python3-mako python3-markupsafe python3-matplotlib 2021-02-11T06:08:37.372 INFO:teuthology.orchestra.run.smithi061.stdout: python3-monotonic python3-numpy python3-oauth2client python3-olefile 2021-02-11T06:08:37.372 INFO:teuthology.orchestra.run.smithi061.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2021-02-11T06:08:37.372 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py 2021-02-11T06:08:37.372 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyparsing 2021-02-11T06:08:37.372 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-rgw 2021-02-11T06:08:37.372 INFO:teuthology.orchestra.run.smithi061.stdout: python3-routes python3-rsa python3-scipy python3-setuptools 2021-02-11T06:08:37.372 INFO:teuthology.orchestra.run.smithi061.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2021-02-11T06:08:37.372 INFO:teuthology.orchestra.run.smithi061.stdout: python3-sklearn python3-sklearn-lib python3-sqlalchemy python3-tempita 2021-02-11T06:08:37.373 INFO:teuthology.orchestra.run.smithi061.stdout: python3-tk python3-tz python3-uritemplate python3-waitress 2021-02-11T06:08:37.373 INFO:teuthology.orchestra.run.smithi061.stdout: python3-webencodings python3-webob python3-websocket python3-webtest 2021-02-11T06:08:37.373 INFO:teuthology.orchestra.run.smithi061.stdout: python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2021-02-11T06:08:37.373 INFO:teuthology.orchestra.run.smithi061.stdout:The following packages will be upgraded: 2021-02-11T06:08:37.373 INFO:teuthology.orchestra.run.smithi061.stdout: librados2 librbd1 2021-02-11T06:08:37.475 INFO:teuthology.orchestra.run.smithi172.stdout:2 upgraded, 111 newly installed, 0 to remove and 19 not upgraded. 2021-02-11T06:08:37.475 INFO:teuthology.orchestra.run.smithi172.stdout:Need to get 166 MB of archives. 2021-02-11T06:08:37.475 INFO:teuthology.orchestra.run.smithi172.stdout:After this operation, 776 MB of additional disk space will be used. 2021-02-11T06:08:37.475 INFO:teuthology.orchestra.run.smithi172.stdout:Get:1 https://1.chacra.ceph.com/r/ceph/wip-easter-bunny/84a8025e5a892a6879f75e17e03444bf443eae4b/ubuntu/bionic/flavors/default bionic/main amd64 libjaeger amd64 17.0.0-711-g84a8025e-1bionic [3,720 B] 2021-02-11T06:08:37.516 INFO:teuthology.orchestra.run.smithi061.stdout:2 upgraded, 111 newly installed, 0 to remove and 19 not upgraded. 2021-02-11T06:08:37.516 INFO:teuthology.orchestra.run.smithi061.stdout:Need to get 166 MB of archives. 2021-02-11T06:08:37.516 INFO:teuthology.orchestra.run.smithi061.stdout:After this operation, 776 MB of additional disk space will be used. 2021-02-11T06:08:37.516 INFO:teuthology.orchestra.run.smithi061.stdout:Get:1 https://1.chacra.ceph.com/r/ceph/wip-easter-bunny/84a8025e5a892a6879f75e17e03444bf443eae4b/ubuntu/bionic/flavors/default bionic/main amd64 libjaeger amd64 17.0.0-711-g84a8025e-1bionic [3,720 B] 2021-02-11T06:08:37.523 INFO:teuthology.orchestra.run.smithi172.stdout:Get:2 https://1.chacra.ceph.com/r/ceph/wip-easter-bunny/84a8025e5a892a6879f75e17e03444bf443eae4b/ubuntu/bionic/flavors/default bionic/main amd64 librbd1 amd64 17.0.0-711-g84a8025e-1bionic [3,122 kB] 2021-02-11T06:08:37.525 INFO:teuthology.orchestra.run.smithi172.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-uritemplate all 0.6-1ubuntu1 [5,126 B] 2021-02-11T06:08:37.563 INFO:teuthology.orchestra.run.smithi061.stdout:Get:2 https://1.chacra.ceph.com/r/ceph/wip-easter-bunny/84a8025e5a892a6879f75e17e03444bf443eae4b/ubuntu/bionic/flavors/default bionic/main amd64 librbd1 amd64 17.0.0-711-g84a8025e-1bionic [3,122 kB] 2021-02-11T06:08:37.565 INFO:teuthology.orchestra.run.smithi061.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-uritemplate all 0.6-1ubuntu1 [5,126 B] 2021-02-11T06:08:37.605 INFO:teuthology.orchestra.run.smithi172.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liblttng-ust0 amd64 2.10.1-1 [154 kB] 2021-02-11T06:08:37.647 INFO:teuthology.orchestra.run.smithi061.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liblttng-ust0 amd64 2.10.1-1 [154 kB] 2021-02-11T06:08:37.749 INFO:teuthology.orchestra.run.smithi172.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/wip-easter-bunny/84a8025e5a892a6879f75e17e03444bf443eae4b/ubuntu/bionic/flavors/default bionic/main amd64 librados2 amd64 17.0.0-711-g84a8025e-1bionic [3,161 kB] 2021-02-11T06:08:37.856 INFO:teuthology.orchestra.run.smithi172.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-prettytable all 0.7.2-3 [19.7 kB] 2021-02-11T06:08:37.871 INFO:teuthology.orchestra.run.smithi172.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 librabbitmq4 amd64 0.8.0-1ubuntu0.18.04.2 [33.9 kB] 2021-02-11T06:08:37.871 INFO:teuthology.orchestra.run.smithi061.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/wip-easter-bunny/84a8025e5a892a6879f75e17e03444bf443eae4b/ubuntu/bionic/flavors/default bionic/main amd64 librados2 amd64 17.0.0-711-g84a8025e-1bionic [3,161 kB] 2021-02-11T06:08:37.895 INFO:teuthology.orchestra.run.smithi172.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic/universe amd64 librdkafka1 amd64 0.11.3-1build1 [293 kB] 2021-02-11T06:08:37.904 INFO:teuthology.orchestra.run.smithi061.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-prettytable all 0.7.2-3 [19.7 kB] 2021-02-11T06:08:37.919 INFO:teuthology.orchestra.run.smithi061.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 librabbitmq4 amd64 0.8.0-1ubuntu0.18.04.2 [33.9 kB] 2021-02-11T06:08:37.931 INFO:teuthology.orchestra.run.smithi172.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/wip-easter-bunny/84a8025e5a892a6879f75e17e03444bf443eae4b/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs2 amd64 17.0.0-711-g84a8025e-1bionic [665 kB] 2021-02-11T06:08:37.937 INFO:teuthology.orchestra.run.smithi061.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic/universe amd64 librdkafka1 amd64 0.11.3-1build1 [293 kB] 2021-02-11T06:08:37.972 INFO:teuthology.orchestra.run.smithi172.stdout:Get:10 https://1.chacra.ceph.com/r/ceph/wip-easter-bunny/84a8025e5a892a6879f75e17e03444bf443eae4b/ubuntu/bionic/flavors/default bionic/main amd64 python3-rados amd64 17.0.0-711-g84a8025e-1bionic [340 kB] 2021-02-11T06:08:37.983 INFO:teuthology.orchestra.run.smithi172.stdout:Get:11 https://1.chacra.ceph.com/r/ceph/wip-easter-bunny/84a8025e5a892a6879f75e17e03444bf443eae4b/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-argparse all 17.0.0-711-g84a8025e-1bionic [21.7 kB] 2021-02-11T06:08:37.984 INFO:teuthology.orchestra.run.smithi172.stdout:Get:12 https://1.chacra.ceph.com/r/ceph/wip-easter-bunny/84a8025e5a892a6879f75e17e03444bf443eae4b/ubuntu/bionic/flavors/default bionic/main amd64 python3-cephfs amd64 17.0.0-711-g84a8025e-1bionic [163 kB] 2021-02-11T06:08:37.993 INFO:teuthology.orchestra.run.smithi172.stdout:Get:13 http://archive.ubuntu.com/ubuntu bionic/main amd64 libsnappy1v5 amd64 1.1.7-1 [16.0 kB] 2021-02-11T06:08:37.997 INFO:teuthology.orchestra.run.smithi172.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic/main amd64 libleveldb1v5 amd64 1.20-2 [136 kB] 2021-02-11T06:08:38.009 INFO:teuthology.orchestra.run.smithi172.stdout:Get:15 https://1.chacra.ceph.com/r/ceph/wip-easter-bunny/84a8025e5a892a6879f75e17e03444bf443eae4b/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-common all 17.0.0-711-g84a8025e-1bionic [29.2 kB] 2021-02-11T06:08:38.011 INFO:teuthology.orchestra.run.smithi172.stdout:Get:16 https://1.chacra.ceph.com/r/ceph/wip-easter-bunny/84a8025e5a892a6879f75e17e03444bf443eae4b/ubuntu/bionic/flavors/default bionic/main amd64 python3-rbd amd64 17.0.0-711-g84a8025e-1bionic [335 kB] 2021-02-11T06:08:38.021 INFO:teuthology.orchestra.run.smithi172.stdout:Get:17 https://1.chacra.ceph.com/r/ceph/wip-easter-bunny/84a8025e5a892a6879f75e17e03444bf443eae4b/ubuntu/bionic/flavors/default bionic/main amd64 librgw2 amd64 17.0.0-711-g84a8025e-1bionic [3,296 kB] 2021-02-11T06:08:38.024 INFO:teuthology.orchestra.run.smithi172.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liboath0 amd64 2.6.1-1 [44.7 kB] 2021-02-11T06:08:38.031 INFO:teuthology.orchestra.run.smithi172.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-dateutil all 2.6.1-1 [52.3 kB] 2021-02-11T06:08:38.038 INFO:teuthology.orchestra.run.smithi061.stdout:Get:9 http://archive.ubuntu.com/ubuntu bionic/main amd64 libsnappy1v5 amd64 1.1.7-1 [16.0 kB] 2021-02-11T06:08:38.040 INFO:teuthology.orchestra.run.smithi172.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-asn1crypto all 0.24.0-1 [72.8 kB] 2021-02-11T06:08:38.042 INFO:teuthology.orchestra.run.smithi061.stdout:Get:10 http://archive.ubuntu.com/ubuntu bionic/main amd64 libleveldb1v5 amd64 1.20-2 [136 kB] 2021-02-11T06:08:38.050 INFO:teuthology.orchestra.run.smithi172.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-cffi-backend amd64 1.11.5-1 [64.6 kB] 2021-02-11T06:08:38.058 INFO:teuthology.orchestra.run.smithi172.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-cryptography amd64 2.1.4-1ubuntu1.4 [220 kB] 2021-02-11T06:08:38.067 INFO:teuthology.orchestra.run.smithi061.stdout:Get:11 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liboath0 amd64 2.6.1-1 [44.7 kB] 2021-02-11T06:08:38.074 INFO:teuthology.orchestra.run.smithi061.stdout:Get:12 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-dateutil all 2.6.1-1 [52.3 kB] 2021-02-11T06:08:38.082 INFO:teuthology.orchestra.run.smithi061.stdout:Get:13 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-asn1crypto all 0.24.0-1 [72.8 kB] 2021-02-11T06:08:38.083 INFO:teuthology.orchestra.run.smithi172.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-openssl all 17.5.0-1ubuntu1 [41.5 kB] 2021-02-11T06:08:38.088 INFO:teuthology.orchestra.run.smithi172.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-bcrypt amd64 3.1.4-2 [29.9 kB] 2021-02-11T06:08:38.095 INFO:teuthology.orchestra.run.smithi061.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-cffi-backend amd64 1.11.5-1 [64.6 kB] 2021-02-11T06:08:38.098 INFO:teuthology.orchestra.run.smithi172.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cherrypy3 all 8.9.1-2 [160 kB] 2021-02-11T06:08:38.101 INFO:teuthology.orchestra.run.smithi061.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-cryptography amd64 2.1.4-1ubuntu1.4 [220 kB] 2021-02-11T06:08:38.111 INFO:teuthology.orchestra.run.smithi172.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jwt all 1.5.3+ds1-1 [15.9 kB] 2021-02-11T06:08:38.129 INFO:teuthology.orchestra.run.smithi061.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-openssl all 17.5.0-1ubuntu1 [41.5 kB] 2021-02-11T06:08:38.132 INFO:teuthology.orchestra.run.smithi061.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-bcrypt amd64 3.1.4-2 [29.9 kB] 2021-02-11T06:08:38.151 INFO:teuthology.orchestra.run.smithi061.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cherrypy3 all 8.9.1-2 [160 kB] 2021-02-11T06:08:38.156 INFO:teuthology.orchestra.run.smithi061.stdout:Get:19 https://1.chacra.ceph.com/r/ceph/wip-easter-bunny/84a8025e5a892a6879f75e17e03444bf443eae4b/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs2 amd64 17.0.0-711-g84a8025e-1bionic [665 kB] 2021-02-11T06:08:38.163 INFO:teuthology.orchestra.run.smithi061.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jwt all 1.5.3+ds1-1 [15.9 kB] 2021-02-11T06:08:38.164 INFO:teuthology.orchestra.run.smithi061.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-logutils all 0.3.3-5 [16.7 kB] 2021-02-11T06:08:38.171 INFO:teuthology.orchestra.run.smithi172.stdout:Get:27 https://1.chacra.ceph.com/r/ceph/wip-easter-bunny/84a8025e5a892a6879f75e17e03444bf443eae4b/ubuntu/bionic/flavors/default bionic/main amd64 python3-rgw amd64 17.0.0-711-g84a8025e-1bionic [99.2 kB] 2021-02-11T06:08:38.174 INFO:teuthology.orchestra.run.smithi172.stdout:Get:28 https://1.chacra.ceph.com/r/ceph/wip-easter-bunny/84a8025e5a892a6879f75e17e03444bf443eae4b/ubuntu/bionic/flavors/default bionic/main amd64 libradosstriper1 amd64 17.0.0-711-g84a8025e-1bionic [414 kB] 2021-02-11T06:08:38.177 INFO:teuthology.orchestra.run.smithi172.stdout:Get:29 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-logutils all 0.3.3-5 [16.7 kB] 2021-02-11T06:08:38.179 INFO:teuthology.orchestra.run.smithi172.stdout:Get:30 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-markupsafe amd64 1.0-1build1 [13.5 kB] 2021-02-11T06:08:38.179 INFO:teuthology.orchestra.run.smithi172.stdout:Get:31 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-mako all 1.0.7+ds1-1 [59.3 kB] 2021-02-11T06:08:38.183 INFO:teuthology.orchestra.run.smithi172.stdout:Get:32 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-simplegeneric all 0.8.1-1 [11.5 kB] 2021-02-11T06:08:38.184 INFO:teuthology.orchestra.run.smithi172.stdout:Get:33 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-singledispatch all 3.4.0.3-2 [7,022 B] 2021-02-11T06:08:38.184 INFO:teuthology.orchestra.run.smithi172.stdout:Get:34 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webob all 1:1.7.3-2fakesync1 [64.3 kB] 2021-02-11T06:08:38.186 INFO:teuthology.orchestra.run.smithi172.stdout:Get:35 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-bs4 all 4.6.0-1 [67.8 kB] 2021-02-11T06:08:38.188 INFO:teuthology.orchestra.run.smithi172.stdout:Get:36 https://1.chacra.ceph.com/r/ceph/wip-easter-bunny/84a8025e5a892a6879f75e17e03444bf443eae4b/ubuntu/bionic/flavors/default bionic/main amd64 ceph-common amd64 17.0.0-711-g84a8025e-1bionic [21.2 MB] 2021-02-11T06:08:38.189 INFO:teuthology.orchestra.run.smithi172.stdout:Get:37 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-waitress all 1.0.1-1 [53.4 kB] 2021-02-11T06:08:38.209 INFO:teuthology.orchestra.run.smithi061.stdout:Get:22 https://1.chacra.ceph.com/r/ceph/wip-easter-bunny/84a8025e5a892a6879f75e17e03444bf443eae4b/ubuntu/bionic/flavors/default bionic/main amd64 python3-rados amd64 17.0.0-711-g84a8025e-1bionic [340 kB] 2021-02-11T06:08:38.231 INFO:teuthology.orchestra.run.smithi061.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-markupsafe amd64 1.0-1build1 [13.5 kB] 2021-02-11T06:08:38.232 INFO:teuthology.orchestra.run.smithi061.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-mako all 1.0.7+ds1-1 [59.3 kB] 2021-02-11T06:08:38.234 INFO:teuthology.orchestra.run.smithi061.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-simplegeneric all 0.8.1-1 [11.5 kB] 2021-02-11T06:08:38.235 INFO:teuthology.orchestra.run.smithi061.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-singledispatch all 3.4.0.3-2 [7,022 B] 2021-02-11T06:08:38.236 INFO:teuthology.orchestra.run.smithi061.stdout:Get:27 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webob all 1:1.7.3-2fakesync1 [64.3 kB] 2021-02-11T06:08:38.238 INFO:teuthology.orchestra.run.smithi061.stdout:Get:28 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-bs4 all 4.6.0-1 [67.8 kB] 2021-02-11T06:08:38.240 INFO:teuthology.orchestra.run.smithi061.stdout:Get:29 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-waitress all 1.0.1-1 [53.4 kB] 2021-02-11T06:08:38.253 INFO:teuthology.orchestra.run.smithi061.stdout:Get:30 https://1.chacra.ceph.com/r/ceph/wip-easter-bunny/84a8025e5a892a6879f75e17e03444bf443eae4b/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-argparse all 17.0.0-711-g84a8025e-1bionic [21.7 kB] 2021-02-11T06:08:39.312 INFO:teuthology.orchestra.run.smithi172.stdout:Get:38 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tempita all 0.5.2-2 [13.9 kB] 2021-02-11T06:08:39.312 INFO:teuthology.orchestra.run.smithi172.stdout:Get:39 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-paste all 2.0.3+dfsg-4ubuntu1 [456 kB] 2021-02-11T06:08:39.312 INFO:teuthology.orchestra.run.smithi172.stdout:Get:40 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy-tpl all 1.5.2-4 [4,796 B] 2021-02-11T06:08:39.312 INFO:teuthology.orchestra.run.smithi172.stdout:Get:41 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastedeploy all 1.5.2-4 [13.4 kB] 2021-02-11T06:08:39.312 INFO:teuthology.orchestra.run.smithi172.stdout:Get:42 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webtest all 2.0.28-1ubuntu1 [27.9 kB] 2021-02-11T06:08:39.312 INFO:teuthology.orchestra.run.smithi172.stdout:Get:43 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pecan all 1.2.1-2 [86.1 kB] 2021-02-11T06:08:39.313 INFO:teuthology.orchestra.run.smithi172.stdout:Get:44 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 python3-werkzeug all 0.14.1+dfsg1-1ubuntu0.1 [174 kB] 2021-02-11T06:08:39.313 INFO:teuthology.orchestra.run.smithi172.stdout:Get:45 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-jinja2 all 2.10-1ubuntu0.18.04.1 [95.4 kB] 2021-02-11T06:08:39.313 INFO:teuthology.orchestra.run.smithi172.stdout:Get:46 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-repoze.lru all 0.7-1 [11.7 kB] 2021-02-11T06:08:39.313 INFO:teuthology.orchestra.run.smithi172.stdout:Get:47 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-routes all 2.4.1-1 [28.4 kB] 2021-02-11T06:08:39.313 INFO:teuthology.orchestra.run.smithi172.stdout:Get:48 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-numpy amd64 1:1.13.3-2ubuntu1 [1,943 kB] 2021-02-11T06:08:39.313 INFO:teuthology.orchestra.run.smithi172.stdout:Get:49 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-decorator all 4.1.2-1 [9,364 B] 2021-02-11T06:08:39.313 INFO:teuthology.orchestra.run.smithi172.stdout:Get:50 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-scipy amd64 0.19.1-2ubuntu1 [9,619 kB] 2021-02-11T06:08:39.313 INFO:teuthology.orchestra.run.smithi172.stdout:Get:51 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn-lib amd64 0.19.1-3 [1,429 kB] 2021-02-11T06:08:39.313 INFO:teuthology.orchestra.run.smithi172.stdout:Get:52 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-joblib all 0.11-1 [109 kB] 2021-02-11T06:08:39.313 INFO:teuthology.orchestra.run.smithi172.stdout:Get:53 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn all 0.19.1-3 [1,373 kB] 2021-02-11T06:08:39.313 INFO:teuthology.orchestra.run.smithi172.stdout:Get:54 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-monotonic all 1.1-2 [5,332 B] 2021-02-11T06:08:39.313 INFO:teuthology.orchestra.run.smithi172.stdout:Get:55 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-fasteners all 0.12.0-3 [13.5 kB] 2021-02-11T06:08:39.313 INFO:teuthology.orchestra.run.smithi172.stdout:Get:56 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-httplib2 all 0.9.2+dfsg-1ubuntu0.2 [28.6 kB] 2021-02-11T06:08:39.314 INFO:teuthology.orchestra.run.smithi172.stdout:Get:57 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1 all 0.4.2-3 [46.8 kB] 2021-02-11T06:08:39.314 INFO:teuthology.orchestra.run.smithi172.stdout:Get:58 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1-modules all 0.2.1-0.2 [32.9 kB] 2021-02-11T06:08:39.314 INFO:teuthology.orchestra.run.smithi172.stdout:Get:59 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-rsa all 3.4.2-1 [29.0 kB] 2021-02-11T06:08:39.314 INFO:teuthology.orchestra.run.smithi172.stdout:Get:60 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-sqlalchemy all 1.1.11+ds1-1ubuntu1 [680 kB] 2021-02-11T06:08:39.314 INFO:teuthology.orchestra.run.smithi172.stdout:Get:61 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-oauth2client all 4.1.2-3 [61.5 kB] 2021-02-11T06:08:39.314 INFO:teuthology.orchestra.run.smithi172.stdout:Get:62 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-setuptools all 39.0.1-2 [248 kB] 2021-02-11T06:08:39.314 INFO:teuthology.orchestra.run.smithi172.stdout:Get:63 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-websocket all 0.44.0-0ubuntu2 [30.7 kB] 2021-02-11T06:08:39.314 INFO:teuthology.orchestra.run.smithi172.stdout:Get:64 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-kubernetes all 2.0.0-2ubuntu1 [181 kB] 2021-02-11T06:08:39.314 INFO:teuthology.orchestra.run.smithi172.stdout:Get:65 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-json-pointer all 1.10-1 [7,478 B] 2021-02-11T06:08:39.314 INFO:teuthology.orchestra.run.smithi172.stdout:Get:66 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jsonpatch all 1.19+really1.16-1fakesync1 [13.4 kB] 2021-02-11T06:08:39.314 INFO:teuthology.orchestra.run.smithi172.stdout:Get:67 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libonig4 amd64 6.7.0-1 [119 kB] 2021-02-11T06:08:39.315 INFO:teuthology.orchestra.run.smithi172.stdout:Get:68 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libjq1 amd64 1.5+dfsg-2 [111 kB] 2021-02-11T06:08:39.315 INFO:teuthology.orchestra.run.smithi172.stdout:Get:69 http://archive.ubuntu.com/ubuntu bionic/universe amd64 jq amd64 1.5+dfsg-2 [45.6 kB] 2021-02-11T06:08:39.315 INFO:teuthology.orchestra.run.smithi172.stdout:Get:70 http://archive.ubuntu.com/ubuntu bionic/main amd64 socat amd64 1.7.3.2-2ubuntu2 [342 kB] 2021-02-11T06:08:39.315 INFO:teuthology.orchestra.run.smithi172.stdout:Get:71 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxslt1.1 amd64 1.1.29-5ubuntu0.2 [150 kB] 2021-02-11T06:08:39.315 INFO:teuthology.orchestra.run.smithi172.stdout:Get:72 http://archive.ubuntu.com/ubuntu bionic/universe amd64 xmlstarlet amd64 1.6.1-2 [382 kB] 2021-02-11T06:08:39.315 INFO:teuthology.orchestra.run.smithi172.stdout:Get:73 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 nvme-cli amd64 1.5-1ubuntu1.1 [184 kB] 2021-02-11T06:08:39.315 INFO:teuthology.orchestra.run.smithi172.stdout:Get:74 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-attr all 17.4.0-2 [23.7 kB] 2021-02-11T06:08:39.315 INFO:teuthology.orchestra.run.smithi172.stdout:Get:75 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-funcsigs all 1.0.2-4 [13.5 kB] 2021-02-11T06:08:39.315 INFO:teuthology.orchestra.run.smithi172.stdout:Get:76 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pluggy all 0.6.0-1 [12.2 kB] 2021-02-11T06:08:39.315 INFO:teuthology.orchestra.run.smithi172.stdout:Get:77 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-py all 1.5.2-1 [65.1 kB] 2021-02-11T06:08:39.315 INFO:teuthology.orchestra.run.smithi172.stdout:Get:78 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pytest all 3.3.2-2 [142 kB] 2021-02-11T06:08:39.315 INFO:teuthology.orchestra.run.smithi172.stdout:Get:79 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-attr all 17.4.0-2 [23.8 kB] 2021-02-11T06:08:39.315 INFO:teuthology.orchestra.run.smithi172.stdout:Get:80 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cycler all 0.10.0-1 [7,622 B] 2021-02-11T06:08:39.315 INFO:teuthology.orchestra.run.smithi172.stdout:Get:81 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webencodings all 0.5-2 [10.4 kB] 2021-02-11T06:08:39.316 INFO:teuthology.orchestra.run.smithi172.stdout:Get:82 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-html5lib all 0.999999999-1 [81.9 kB] 2021-02-11T06:08:39.316 INFO:teuthology.orchestra.run.smithi172.stdout:Get:83 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-lxml amd64 4.2.1-1ubuntu0.3 [1,097 kB] 2021-02-11T06:08:39.316 INFO:teuthology.orchestra.run.smithi172.stdout:Get:84 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyparsing all 2.2.0+dfsg1-2 [52.2 kB] 2021-02-11T06:08:39.316 INFO:teuthology.orchestra.run.smithi172.stdout:Get:85 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tz all 2018.3-2 [25.1 kB] 2021-02-11T06:08:39.316 INFO:teuthology.orchestra.run.smithi172.stdout:Get:86 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-matplotlib amd64 2.1.1-2ubuntu3 [3,907 kB] 2021-02-11T06:08:39.316 INFO:teuthology.orchestra.run.smithi061.stdout:Get:31 https://1.chacra.ceph.com/r/ceph/wip-easter-bunny/84a8025e5a892a6879f75e17e03444bf443eae4b/ubuntu/bionic/flavors/default bionic/main amd64 python3-cephfs amd64 17.0.0-711-g84a8025e-1bionic [163 kB] 2021-02-11T06:08:39.316 INFO:teuthology.orchestra.run.smithi061.stdout:Get:32 https://1.chacra.ceph.com/r/ceph/wip-easter-bunny/84a8025e5a892a6879f75e17e03444bf443eae4b/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-common all 17.0.0-711-g84a8025e-1bionic [29.2 kB] 2021-02-11T06:08:39.316 INFO:teuthology.orchestra.run.smithi061.stdout:Get:33 https://1.chacra.ceph.com/r/ceph/wip-easter-bunny/84a8025e5a892a6879f75e17e03444bf443eae4b/ubuntu/bionic/flavors/default bionic/main amd64 python3-rbd amd64 17.0.0-711-g84a8025e-1bionic [335 kB] 2021-02-11T06:08:39.316 INFO:teuthology.orchestra.run.smithi061.stdout:Get:34 https://1.chacra.ceph.com/r/ceph/wip-easter-bunny/84a8025e5a892a6879f75e17e03444bf443eae4b/ubuntu/bionic/flavors/default bionic/main amd64 librgw2 amd64 17.0.0-711-g84a8025e-1bionic [3,296 kB] 2021-02-11T06:08:39.316 INFO:teuthology.orchestra.run.smithi061.stdout:Get:35 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tempita all 0.5.2-2 [13.9 kB] 2021-02-11T06:08:39.316 INFO:teuthology.orchestra.run.smithi061.stdout:Get:36 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-paste all 2.0.3+dfsg-4ubuntu1 [456 kB] 2021-02-11T06:08:39.316 INFO:teuthology.orchestra.run.smithi061.stdout:Get:37 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy-tpl all 1.5.2-4 [4,796 B] 2021-02-11T06:08:39.316 INFO:teuthology.orchestra.run.smithi061.stdout:Get:38 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastedeploy all 1.5.2-4 [13.4 kB] 2021-02-11T06:08:39.317 INFO:teuthology.orchestra.run.smithi061.stdout:Get:39 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webtest all 2.0.28-1ubuntu1 [27.9 kB] 2021-02-11T06:08:39.317 INFO:teuthology.orchestra.run.smithi061.stdout:Get:40 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pecan all 1.2.1-2 [86.1 kB] 2021-02-11T06:08:39.317 INFO:teuthology.orchestra.run.smithi061.stdout:Get:41 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 python3-werkzeug all 0.14.1+dfsg1-1ubuntu0.1 [174 kB] 2021-02-11T06:08:39.317 INFO:teuthology.orchestra.run.smithi061.stdout:Get:42 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-jinja2 all 2.10-1ubuntu0.18.04.1 [95.4 kB] 2021-02-11T06:08:39.317 INFO:teuthology.orchestra.run.smithi061.stdout:Get:43 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-repoze.lru all 0.7-1 [11.7 kB] 2021-02-11T06:08:39.317 INFO:teuthology.orchestra.run.smithi061.stdout:Get:44 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-routes all 2.4.1-1 [28.4 kB] 2021-02-11T06:08:39.317 INFO:teuthology.orchestra.run.smithi061.stdout:Get:45 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-numpy amd64 1:1.13.3-2ubuntu1 [1,943 kB] 2021-02-11T06:08:39.317 INFO:teuthology.orchestra.run.smithi061.stdout:Get:46 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-decorator all 4.1.2-1 [9,364 B] 2021-02-11T06:08:39.317 INFO:teuthology.orchestra.run.smithi061.stdout:Get:47 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-scipy amd64 0.19.1-2ubuntu1 [9,619 kB] 2021-02-11T06:08:39.317 INFO:teuthology.orchestra.run.smithi061.stdout:Get:48 https://1.chacra.ceph.com/r/ceph/wip-easter-bunny/84a8025e5a892a6879f75e17e03444bf443eae4b/ubuntu/bionic/flavors/default bionic/main amd64 python3-rgw amd64 17.0.0-711-g84a8025e-1bionic [99.2 kB] 2021-02-11T06:08:39.317 INFO:teuthology.orchestra.run.smithi061.stdout:Get:49 https://1.chacra.ceph.com/r/ceph/wip-easter-bunny/84a8025e5a892a6879f75e17e03444bf443eae4b/ubuntu/bionic/flavors/default bionic/main amd64 libradosstriper1 amd64 17.0.0-711-g84a8025e-1bionic [414 kB] 2021-02-11T06:08:39.317 INFO:teuthology.orchestra.run.smithi061.stdout:Get:50 https://1.chacra.ceph.com/r/ceph/wip-easter-bunny/84a8025e5a892a6879f75e17e03444bf443eae4b/ubuntu/bionic/flavors/default bionic/main amd64 ceph-common amd64 17.0.0-711-g84a8025e-1bionic [21.2 MB] 2021-02-11T06:08:39.317 INFO:teuthology.orchestra.run.smithi061.stdout:Get:51 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn-lib amd64 0.19.1-3 [1,429 kB] 2021-02-11T06:08:39.318 INFO:teuthology.orchestra.run.smithi061.stdout:Get:52 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-joblib all 0.11-1 [109 kB] 2021-02-11T06:08:39.318 INFO:teuthology.orchestra.run.smithi061.stdout:Get:53 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn all 0.19.1-3 [1,373 kB] 2021-02-11T06:08:39.318 INFO:teuthology.orchestra.run.smithi061.stdout:Get:54 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-monotonic all 1.1-2 [5,332 B] 2021-02-11T06:08:39.318 INFO:teuthology.orchestra.run.smithi061.stdout:Get:55 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-fasteners all 0.12.0-3 [13.5 kB] 2021-02-11T06:08:39.318 INFO:teuthology.orchestra.run.smithi061.stdout:Get:56 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-httplib2 all 0.9.2+dfsg-1ubuntu0.2 [28.6 kB] 2021-02-11T06:08:39.318 INFO:teuthology.orchestra.run.smithi061.stdout:Get:57 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1 all 0.4.2-3 [46.8 kB] 2021-02-11T06:08:39.318 INFO:teuthology.orchestra.run.smithi061.stdout:Get:58 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1-modules all 0.2.1-0.2 [32.9 kB] 2021-02-11T06:08:39.318 INFO:teuthology.orchestra.run.smithi061.stdout:Get:59 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-rsa all 3.4.2-1 [29.0 kB] 2021-02-11T06:08:39.318 INFO:teuthology.orchestra.run.smithi061.stdout:Get:60 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-sqlalchemy all 1.1.11+ds1-1ubuntu1 [680 kB] 2021-02-11T06:08:39.318 INFO:teuthology.orchestra.run.smithi061.stdout:Get:61 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-oauth2client all 4.1.2-3 [61.5 kB] 2021-02-11T06:08:39.318 INFO:teuthology.orchestra.run.smithi061.stdout:Get:62 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-setuptools all 39.0.1-2 [248 kB] 2021-02-11T06:08:39.319 INFO:teuthology.orchestra.run.smithi061.stdout:Get:63 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-websocket all 0.44.0-0ubuntu2 [30.7 kB] 2021-02-11T06:08:39.319 INFO:teuthology.orchestra.run.smithi061.stdout:Get:64 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-kubernetes all 2.0.0-2ubuntu1 [181 kB] 2021-02-11T06:08:39.319 INFO:teuthology.orchestra.run.smithi061.stdout:Get:65 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-json-pointer all 1.10-1 [7,478 B] 2021-02-11T06:08:39.319 INFO:teuthology.orchestra.run.smithi061.stdout:Get:66 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jsonpatch all 1.19+really1.16-1fakesync1 [13.4 kB] 2021-02-11T06:08:39.319 INFO:teuthology.orchestra.run.smithi061.stdout:Get:67 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libonig4 amd64 6.7.0-1 [119 kB] 2021-02-11T06:08:39.319 INFO:teuthology.orchestra.run.smithi061.stdout:Get:68 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libjq1 amd64 1.5+dfsg-2 [111 kB] 2021-02-11T06:08:39.319 INFO:teuthology.orchestra.run.smithi061.stdout:Get:69 http://archive.ubuntu.com/ubuntu bionic/universe amd64 jq amd64 1.5+dfsg-2 [45.6 kB] 2021-02-11T06:08:39.319 INFO:teuthology.orchestra.run.smithi061.stdout:Get:70 http://archive.ubuntu.com/ubuntu bionic/main amd64 socat amd64 1.7.3.2-2ubuntu2 [342 kB] 2021-02-11T06:08:39.319 INFO:teuthology.orchestra.run.smithi061.stdout:Get:71 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxslt1.1 amd64 1.1.29-5ubuntu0.2 [150 kB] 2021-02-11T06:08:39.319 INFO:teuthology.orchestra.run.smithi061.stdout:Get:72 http://archive.ubuntu.com/ubuntu bionic/universe amd64 xmlstarlet amd64 1.6.1-2 [382 kB] 2021-02-11T06:08:39.319 INFO:teuthology.orchestra.run.smithi061.stdout:Get:73 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 nvme-cli amd64 1.5-1ubuntu1.1 [184 kB] 2021-02-11T06:08:39.319 INFO:teuthology.orchestra.run.smithi061.stdout:Get:74 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-attr all 17.4.0-2 [23.7 kB] 2021-02-11T06:08:39.319 INFO:teuthology.orchestra.run.smithi061.stdout:Get:75 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-funcsigs all 1.0.2-4 [13.5 kB] 2021-02-11T06:08:39.319 INFO:teuthology.orchestra.run.smithi061.stdout:Get:76 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pluggy all 0.6.0-1 [12.2 kB] 2021-02-11T06:08:39.319 INFO:teuthology.orchestra.run.smithi061.stdout:Get:77 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-py all 1.5.2-1 [65.1 kB] 2021-02-11T06:08:39.320 INFO:teuthology.orchestra.run.smithi061.stdout:Get:78 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pytest all 3.3.2-2 [142 kB] 2021-02-11T06:08:39.320 INFO:teuthology.orchestra.run.smithi061.stdout:Get:79 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-attr all 17.4.0-2 [23.8 kB] 2021-02-11T06:08:39.320 INFO:teuthology.orchestra.run.smithi061.stdout:Get:80 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cycler all 0.10.0-1 [7,622 B] 2021-02-11T06:08:39.320 INFO:teuthology.orchestra.run.smithi061.stdout:Get:81 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webencodings all 0.5-2 [10.4 kB] 2021-02-11T06:08:39.320 INFO:teuthology.orchestra.run.smithi061.stdout:Get:82 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-html5lib all 0.999999999-1 [81.9 kB] 2021-02-11T06:08:39.320 INFO:teuthology.orchestra.run.smithi061.stdout:Get:83 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-lxml amd64 4.2.1-1ubuntu0.3 [1,097 kB] 2021-02-11T06:08:39.320 INFO:teuthology.orchestra.run.smithi061.stdout:Get:84 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyparsing all 2.2.0+dfsg1-2 [52.2 kB] 2021-02-11T06:08:39.320 INFO:teuthology.orchestra.run.smithi061.stdout:Get:85 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tz all 2018.3-2 [25.1 kB] 2021-02-11T06:08:39.320 INFO:teuthology.orchestra.run.smithi061.stdout:Get:86 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-matplotlib amd64 2.1.1-2ubuntu3 [3,907 kB] 2021-02-11T06:08:39.323 INFO:teuthology.orchestra.run.smithi061.stdout:Get:87 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-olefile all 0.45.1-1 [33.3 kB] 2021-02-11T06:08:39.324 INFO:teuthology.orchestra.run.smithi061.stdout:Get:88 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastescript all 2.0.2-2 [54.6 kB] 2021-02-11T06:08:39.326 INFO:teuthology.orchestra.run.smithi061.stdout:Get:89 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-pil amd64 5.1.0-1ubuntu0.4 [329 kB] 2021-02-11T06:08:39.334 INFO:teuthology.orchestra.run.smithi061.stdout:Get:90 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pluggy all 0.6.0-1 [12.3 kB] 2021-02-11T06:08:39.335 INFO:teuthology.orchestra.run.smithi061.stdout:Get:91 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-py all 1.5.2-1 [65.2 kB] 2021-02-11T06:08:39.335 INFO:teuthology.orchestra.run.smithi061.stdout:Get:92 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyinotify all 0.9.6-1 [24.7 kB] 2021-02-11T06:08:39.335 INFO:teuthology.orchestra.run.smithi061.stdout:Get:93 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pytest all 3.3.2-2 [142 kB] 2021-02-11T06:08:39.337 INFO:teuthology.orchestra.run.smithi061.stdout:Get:94 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-simplejson amd64 3.13.2-1 [49.6 kB] 2021-02-11T06:08:39.338 INFO:teuthology.orchestra.run.smithi061.stdout:Get:95 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-tk amd64 3.6.9-1~18.04 [119 kB] 2021-02-11T06:08:39.422 INFO:teuthology.orchestra.run.smithi172.stdout:Get:87 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-olefile all 0.45.1-1 [33.3 kB] 2021-02-11T06:08:39.423 INFO:teuthology.orchestra.run.smithi172.stdout:Get:88 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastescript all 2.0.2-2 [54.6 kB] 2021-02-11T06:08:39.424 INFO:teuthology.orchestra.run.smithi172.stdout:Get:89 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-pil amd64 5.1.0-1ubuntu0.4 [329 kB] 2021-02-11T06:08:39.430 INFO:teuthology.orchestra.run.smithi172.stdout:Get:90 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pluggy all 0.6.0-1 [12.3 kB] 2021-02-11T06:08:39.431 INFO:teuthology.orchestra.run.smithi172.stdout:Get:91 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-py all 1.5.2-1 [65.2 kB] 2021-02-11T06:08:39.432 INFO:teuthology.orchestra.run.smithi172.stdout:Get:92 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyinotify all 0.9.6-1 [24.7 kB] 2021-02-11T06:08:39.432 INFO:teuthology.orchestra.run.smithi172.stdout:Get:93 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pytest all 3.3.2-2 [142 kB] 2021-02-11T06:08:39.433 INFO:teuthology.orchestra.run.smithi172.stdout:Get:94 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-simplejson amd64 3.13.2-1 [49.6 kB] 2021-02-11T06:08:39.434 INFO:teuthology.orchestra.run.smithi172.stdout:Get:95 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-tk amd64 3.6.9-1~18.04 [119 kB] 2021-02-11T06:08:39.500 INFO:teuthology.orchestra.run.smithi172.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/wip-easter-bunny/84a8025e5a892a6879f75e17e03444bf443eae4b/ubuntu/bionic/flavors/default bionic/main amd64 ceph-base amd64 17.0.0-711-g84a8025e-1bionic [5,624 kB] 2021-02-11T06:08:39.743 INFO:teuthology.orchestra.run.smithi172.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/wip-easter-bunny/84a8025e5a892a6879f75e17e03444bf443eae4b/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-modules-core all 17.0.0-711-g84a8025e-1bionic [185 kB] 2021-02-11T06:08:39.774 INFO:teuthology.orchestra.run.smithi172.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/wip-easter-bunny/84a8025e5a892a6879f75e17e03444bf443eae4b/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr amd64 17.0.0-711-g84a8025e-1bionic [1,365 kB] 2021-02-11T06:08:39.846 INFO:teuthology.orchestra.run.smithi172.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/wip-easter-bunny/84a8025e5a892a6879f75e17e03444bf443eae4b/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mon amd64 17.0.0-711-g84a8025e-1bionic [6,661 kB] 2021-02-11T06:08:40.203 INFO:teuthology.orchestra.run.smithi172.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/wip-easter-bunny/84a8025e5a892a6879f75e17e03444bf443eae4b/ubuntu/bionic/flavors/default bionic/main amd64 ceph-osd amd64 17.0.0-711-g84a8025e-1bionic [24.9 MB] 2021-02-11T06:08:40.714 INFO:teuthology.orchestra.run.smithi061.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/wip-easter-bunny/84a8025e5a892a6879f75e17e03444bf443eae4b/ubuntu/bionic/flavors/default bionic/main amd64 ceph-base amd64 17.0.0-711-g84a8025e-1bionic [5,624 kB] 2021-02-11T06:08:41.154 INFO:teuthology.orchestra.run.smithi061.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/wip-easter-bunny/84a8025e5a892a6879f75e17e03444bf443eae4b/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-modules-core all 17.0.0-711-g84a8025e-1bionic [185 kB] 2021-02-11T06:08:41.159 INFO:teuthology.orchestra.run.smithi061.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/wip-easter-bunny/84a8025e5a892a6879f75e17e03444bf443eae4b/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr amd64 17.0.0-711-g84a8025e-1bionic [1,365 kB] 2021-02-11T06:08:41.289 INFO:teuthology.orchestra.run.smithi061.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/wip-easter-bunny/84a8025e5a892a6879f75e17e03444bf443eae4b/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mon amd64 17.0.0-711-g84a8025e-1bionic [6,661 kB] 2021-02-11T06:08:41.592 INFO:teuthology.orchestra.run.smithi172.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/wip-easter-bunny/84a8025e5a892a6879f75e17e03444bf443eae4b/ubuntu/bionic/flavors/default bionic/main amd64 ceph amd64 17.0.0-711-g84a8025e-1bionic [3,812 B] 2021-02-11T06:08:42.534 INFO:teuthology.orchestra.run.smithi172.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/wip-easter-bunny/84a8025e5a892a6879f75e17e03444bf443eae4b/ubuntu/bionic/flavors/default bionic/main amd64 ceph-fuse amd64 17.0.0-711-g84a8025e-1bionic [770 kB] 2021-02-11T06:08:42.535 INFO:teuthology.orchestra.run.smithi061.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/wip-easter-bunny/84a8025e5a892a6879f75e17e03444bf443eae4b/ubuntu/bionic/flavors/default bionic/main amd64 ceph-osd amd64 17.0.0-711-g84a8025e-1bionic [24.9 MB] 2021-02-11T06:08:42.536 INFO:teuthology.orchestra.run.smithi172.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/wip-easter-bunny/84a8025e5a892a6879f75e17e03444bf443eae4b/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mds amd64 17.0.0-711-g84a8025e-1bionic [2,153 kB] 2021-02-11T06:08:42.536 INFO:teuthology.orchestra.run.smithi172.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/wip-easter-bunny/84a8025e5a892a6879f75e17e03444bf443eae4b/ubuntu/bionic/flavors/default bionic/main amd64 cephadm amd64 17.0.0-711-g84a8025e-1bionic [62.1 kB] 2021-02-11T06:08:42.536 INFO:teuthology.orchestra.run.smithi172.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/wip-easter-bunny/84a8025e5a892a6879f75e17e03444bf443eae4b/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-cephadm all 17.0.0-711-g84a8025e-1bionic [67.2 kB] 2021-02-11T06:08:42.536 INFO:teuthology.orchestra.run.smithi172.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/wip-easter-bunny/84a8025e5a892a6879f75e17e03444bf443eae4b/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-dashboard all 17.0.0-711-g84a8025e-1bionic [2,328 kB] 2021-02-11T06:08:42.536 INFO:teuthology.orchestra.run.smithi172.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/wip-easter-bunny/84a8025e5a892a6879f75e17e03444bf443eae4b/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-local all 17.0.0-711-g84a8025e-1bionic [7,872 kB] 2021-02-11T06:08:42.536 INFO:teuthology.orchestra.run.smithi172.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/wip-easter-bunny/84a8025e5a892a6879f75e17e03444bf443eae4b/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-k8sevents all 17.0.0-711-g84a8025e-1bionic [3,924 B] 2021-02-11T06:08:42.536 INFO:teuthology.orchestra.run.smithi172.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/wip-easter-bunny/84a8025e5a892a6879f75e17e03444bf443eae4b/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-rook all 17.0.0-711-g84a8025e-1bionic [23.3 kB] 2021-02-11T06:08:42.536 INFO:teuthology.orchestra.run.smithi172.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/wip-easter-bunny/84a8025e5a892a6879f75e17e03444bf443eae4b/ubuntu/bionic/flavors/default bionic/main amd64 ceph-test amd64 17.0.0-711-g84a8025e-1bionic [45.0 MB] 2021-02-11T06:08:50.894 INFO:teuthology.orchestra.run.smithi061.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/wip-easter-bunny/84a8025e5a892a6879f75e17e03444bf443eae4b/ubuntu/bionic/flavors/default bionic/main amd64 ceph amd64 17.0.0-711-g84a8025e-1bionic [3,812 B] 2021-02-11T06:08:50.894 INFO:teuthology.orchestra.run.smithi061.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/wip-easter-bunny/84a8025e5a892a6879f75e17e03444bf443eae4b/ubuntu/bionic/flavors/default bionic/main amd64 ceph-fuse amd64 17.0.0-711-g84a8025e-1bionic [770 kB] 2021-02-11T06:08:50.952 INFO:teuthology.orchestra.run.smithi061.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/wip-easter-bunny/84a8025e5a892a6879f75e17e03444bf443eae4b/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mds amd64 17.0.0-711-g84a8025e-1bionic [2,153 kB] 2021-02-11T06:08:51.096 INFO:teuthology.orchestra.run.smithi061.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/wip-easter-bunny/84a8025e5a892a6879f75e17e03444bf443eae4b/ubuntu/bionic/flavors/default bionic/main amd64 cephadm amd64 17.0.0-711-g84a8025e-1bionic [62.1 kB] 2021-02-11T06:08:51.099 INFO:teuthology.orchestra.run.smithi061.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/wip-easter-bunny/84a8025e5a892a6879f75e17e03444bf443eae4b/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-cephadm all 17.0.0-711-g84a8025e-1bionic [67.2 kB] 2021-02-11T06:08:51.101 INFO:teuthology.orchestra.run.smithi061.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/wip-easter-bunny/84a8025e5a892a6879f75e17e03444bf443eae4b/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-dashboard all 17.0.0-711-g84a8025e-1bionic [2,328 kB] 2021-02-11T06:08:51.183 INFO:teuthology.orchestra.run.smithi172.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/wip-easter-bunny/84a8025e5a892a6879f75e17e03444bf443eae4b/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs-dev amd64 17.0.0-711-g84a8025e-1bionic [19.8 kB] 2021-02-11T06:08:51.183 INFO:teuthology.orchestra.run.smithi172.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/wip-easter-bunny/84a8025e5a892a6879f75e17e03444bf443eae4b/ubuntu/bionic/flavors/default bionic/main amd64 radosgw amd64 17.0.0-711-g84a8025e-1bionic [10.1 MB] 2021-02-11T06:08:51.533 INFO:teuthology.orchestra.run.smithi061.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/wip-easter-bunny/84a8025e5a892a6879f75e17e03444bf443eae4b/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-local all 17.0.0-711-g84a8025e-1bionic [7,872 kB] 2021-02-11T06:08:51.775 INFO:teuthology.orchestra.run.smithi172.stdout:Get:113 https://1.chacra.ceph.com/r/ceph/wip-easter-bunny/84a8025e5a892a6879f75e17e03444bf443eae4b/ubuntu/bionic/flavors/default bionic/main amd64 rbd-fuse amd64 17.0.0-711-g84a8025e-1bionic [77.6 kB] 2021-02-11T06:08:51.933 INFO:teuthology.orchestra.run.smithi061.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/wip-easter-bunny/84a8025e5a892a6879f75e17e03444bf443eae4b/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-k8sevents all 17.0.0-711-g84a8025e-1bionic [3,924 B] 2021-02-11T06:08:58.638 INFO:teuthology.orchestra.run.smithi061.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/wip-easter-bunny/84a8025e5a892a6879f75e17e03444bf443eae4b/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-rook all 17.0.0-711-g84a8025e-1bionic [23.3 kB] 2021-02-11T06:08:58.646 INFO:teuthology.orchestra.run.smithi172.stdout:Fetched 166 MB in 14s (11.5 MB/s) 2021-02-11T06:08:58.646 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-uritemplate. 2021-02-11T06:08:58.646 INFO:teuthology.orchestra.run.smithi172.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 137611 files and directories currently installed.) 2021-02-11T06:08:58.646 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../000-python3-uritemplate_0.6-1ubuntu1_all.deb ... 2021-02-11T06:08:58.646 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-uritemplate (0.6-1ubuntu1) ... 2021-02-11T06:08:58.646 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libjaeger. 2021-02-11T06:08:58.646 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../001-libjaeger_17.0.0-711-g84a8025e-1bionic_amd64.deb ... 2021-02-11T06:08:58.646 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libjaeger (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:08:58.646 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2021-02-11T06:08:58.646 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../002-liblttng-ust0_2.10.1-1_amd64.deb ... 2021-02-11T06:08:58.647 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking liblttng-ust0:amd64 (2.10.1-1) ... 2021-02-11T06:08:58.647 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../003-librbd1_17.0.0-711-g84a8025e-1bionic_amd64.deb ... 2021-02-11T06:08:58.647 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking librbd1 (17.0.0-711-g84a8025e-1bionic) over (12.2.13-0ubuntu0.18.04.6) ... 2021-02-11T06:08:58.647 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../004-librados2_17.0.0-711-g84a8025e-1bionic_amd64.deb ... 2021-02-11T06:08:58.647 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking librados2 (17.0.0-711-g84a8025e-1bionic) over (12.2.13-0ubuntu0.18.04.6) ... 2021-02-11T06:08:58.647 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libcephfs2. 2021-02-11T06:08:58.647 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../005-libcephfs2_17.0.0-711-g84a8025e-1bionic_amd64.deb ... 2021-02-11T06:08:58.647 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libcephfs2 (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:08:58.647 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-rados. 2021-02-11T06:08:58.647 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../006-python3-rados_17.0.0-711-g84a8025e-1bionic_amd64.deb ... 2021-02-11T06:08:58.647 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-rados (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:08:58.647 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-ceph-argparse. 2021-02-11T06:08:58.647 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../007-python3-ceph-argparse_17.0.0-711-g84a8025e-1bionic_all.deb ... 2021-02-11T06:08:58.647 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-ceph-argparse (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:08:58.648 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-cephfs. 2021-02-11T06:08:58.648 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../008-python3-cephfs_17.0.0-711-g84a8025e-1bionic_amd64.deb ... 2021-02-11T06:08:58.648 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-cephfs (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:08:58.648 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-ceph-common. 2021-02-11T06:08:58.648 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../009-python3-ceph-common_17.0.0-711-g84a8025e-1bionic_all.deb ... 2021-02-11T06:08:58.648 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-ceph-common (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:08:58.648 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-prettytable. 2021-02-11T06:08:58.648 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../010-python3-prettytable_0.7.2-3_all.deb ... 2021-02-11T06:08:58.648 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-prettytable (0.7.2-3) ... 2021-02-11T06:08:58.648 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-rbd. 2021-02-11T06:08:58.648 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../011-python3-rbd_17.0.0-711-g84a8025e-1bionic_amd64.deb ... 2021-02-11T06:08:58.648 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-rbd (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:08:58.648 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package librabbitmq4:amd64. 2021-02-11T06:08:58.648 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../012-librabbitmq4_0.8.0-1ubuntu0.18.04.2_amd64.deb ... 2021-02-11T06:08:58.649 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2021-02-11T06:08:58.649 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package librdkafka1:amd64. 2021-02-11T06:08:58.649 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../013-librdkafka1_0.11.3-1build1_amd64.deb ... 2021-02-11T06:08:58.649 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking librdkafka1:amd64 (0.11.3-1build1) ... 2021-02-11T06:08:58.649 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package librgw2. 2021-02-11T06:08:58.649 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../014-librgw2_17.0.0-711-g84a8025e-1bionic_amd64.deb ... 2021-02-11T06:08:58.649 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking librgw2 (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:08:58.649 INFO:teuthology.orchestra.run.smithi061.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/wip-easter-bunny/84a8025e5a892a6879f75e17e03444bf443eae4b/ubuntu/bionic/flavors/default bionic/main amd64 ceph-test amd64 17.0.0-711-g84a8025e-1bionic [45.0 MB] 2021-02-11T06:08:58.649 INFO:teuthology.orchestra.run.smithi061.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/wip-easter-bunny/84a8025e5a892a6879f75e17e03444bf443eae4b/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs-dev amd64 17.0.0-711-g84a8025e-1bionic [19.8 kB] 2021-02-11T06:08:58.649 INFO:teuthology.orchestra.run.smithi061.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/wip-easter-bunny/84a8025e5a892a6879f75e17e03444bf443eae4b/ubuntu/bionic/flavors/default bionic/main amd64 radosgw amd64 17.0.0-711-g84a8025e-1bionic [10.1 MB] 2021-02-11T06:08:58.649 INFO:teuthology.orchestra.run.smithi061.stdout:Get:113 https://1.chacra.ceph.com/r/ceph/wip-easter-bunny/84a8025e5a892a6879f75e17e03444bf443eae4b/ubuntu/bionic/flavors/default bionic/main amd64 rbd-fuse amd64 17.0.0-711-g84a8025e-1bionic [77.6 kB] 2021-02-11T06:08:58.649 INFO:teuthology.orchestra.run.smithi061.stdout:Fetched 166 MB in 16s (10.2 MB/s) 2021-02-11T06:08:58.650 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-uritemplate. 2021-02-11T06:08:58.650 INFO:teuthology.orchestra.run.smithi061.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 137611 files and directories currently installed.) 2021-02-11T06:08:58.650 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../000-python3-uritemplate_0.6-1ubuntu1_all.deb ... 2021-02-11T06:08:58.650 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-uritemplate (0.6-1ubuntu1) ... 2021-02-11T06:08:58.650 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package libjaeger. 2021-02-11T06:08:58.650 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../001-libjaeger_17.0.0-711-g84a8025e-1bionic_amd64.deb ... 2021-02-11T06:08:58.650 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking libjaeger (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:08:58.650 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2021-02-11T06:08:58.650 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../002-liblttng-ust0_2.10.1-1_amd64.deb ... 2021-02-11T06:08:58.650 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking liblttng-ust0:amd64 (2.10.1-1) ... 2021-02-11T06:08:58.650 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../003-librbd1_17.0.0-711-g84a8025e-1bionic_amd64.deb ... 2021-02-11T06:08:58.650 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking librbd1 (17.0.0-711-g84a8025e-1bionic) over (12.2.13-0ubuntu0.18.04.6) ... 2021-02-11T06:08:58.650 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../004-librados2_17.0.0-711-g84a8025e-1bionic_amd64.deb ... 2021-02-11T06:08:58.650 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking librados2 (17.0.0-711-g84a8025e-1bionic) over (12.2.13-0ubuntu0.18.04.6) ... 2021-02-11T06:08:58.651 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package libcephfs2. 2021-02-11T06:08:58.651 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../005-libcephfs2_17.0.0-711-g84a8025e-1bionic_amd64.deb ... 2021-02-11T06:08:58.651 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking libcephfs2 (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:08:58.651 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-rados. 2021-02-11T06:08:58.651 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../006-python3-rados_17.0.0-711-g84a8025e-1bionic_amd64.deb ... 2021-02-11T06:08:58.651 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-rados (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:08:58.651 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-ceph-argparse. 2021-02-11T06:08:58.651 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../007-python3-ceph-argparse_17.0.0-711-g84a8025e-1bionic_all.deb ... 2021-02-11T06:08:58.651 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-ceph-argparse (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:08:58.651 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-cephfs. 2021-02-11T06:08:58.651 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../008-python3-cephfs_17.0.0-711-g84a8025e-1bionic_amd64.deb ... 2021-02-11T06:08:58.651 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-cephfs (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:08:58.794 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-ceph-common. 2021-02-11T06:08:58.807 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../009-python3-ceph-common_17.0.0-711-g84a8025e-1bionic_all.deb ... 2021-02-11T06:08:58.844 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-ceph-common (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:08:59.147 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-prettytable. 2021-02-11T06:08:59.157 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-rgw. 2021-02-11T06:08:59.161 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../010-python3-prettytable_0.7.2-3_all.deb ... 2021-02-11T06:08:59.171 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../015-python3-rgw_17.0.0-711-g84a8025e-1bionic_amd64.deb ... 2021-02-11T06:08:59.205 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-prettytable (0.7.2-3) ... 2021-02-11T06:08:59.207 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-rgw (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:08:59.441 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-rbd. 2021-02-11T06:08:59.455 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../011-python3-rbd_17.0.0-711-g84a8025e-1bionic_amd64.deb ... 2021-02-11T06:08:59.468 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libsnappy1v5:amd64. 2021-02-11T06:08:59.482 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../016-libsnappy1v5_1.1.7-1_amd64.deb ... 2021-02-11T06:08:59.499 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-rbd (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:08:59.518 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libsnappy1v5:amd64 (1.1.7-1) ... 2021-02-11T06:08:59.838 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libleveldb1v5:amd64. 2021-02-11T06:08:59.853 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../017-libleveldb1v5_1.20-2_amd64.deb ... 2021-02-11T06:08:59.860 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package librabbitmq4:amd64. 2021-02-11T06:08:59.874 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../012-librabbitmq4_0.8.0-1ubuntu0.18.04.2_amd64.deb ... 2021-02-11T06:08:59.896 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libleveldb1v5:amd64 (1.20-2) ... 2021-02-11T06:08:59.918 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2021-02-11T06:09:00.238 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package librdkafka1:amd64. 2021-02-11T06:09:00.665 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package liboath0. 2021-02-11T06:09:00.665 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../018-liboath0_2.6.1-1_amd64.deb ... 2021-02-11T06:09:00.665 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking liboath0 (2.6.1-1) ... 2021-02-11T06:09:00.665 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libradosstriper1. 2021-02-11T06:09:00.666 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../013-librdkafka1_0.11.3-1build1_amd64.deb ... 2021-02-11T06:09:00.666 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking librdkafka1:amd64 (0.11.3-1build1) ... 2021-02-11T06:09:00.666 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package librgw2. 2021-02-11T06:09:00.666 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../014-librgw2_17.0.0-711-g84a8025e-1bionic_amd64.deb ... 2021-02-11T06:09:00.673 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../019-libradosstriper1_17.0.0-711-g84a8025e-1bionic_amd64.deb ... 2021-02-11T06:09:00.708 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking librgw2 (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:09:00.710 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libradosstriper1 (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:09:01.238 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-common. 2021-02-11T06:09:01.251 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../020-ceph-common_17.0.0-711-g84a8025e-1bionic_amd64.deb ... 2021-02-11T06:09:01.285 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-rgw. 2021-02-11T06:09:01.296 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-common (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:09:01.300 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../015-python3-rgw_17.0.0-711-g84a8025e-1bionic_amd64.deb ... 2021-02-11T06:09:01.343 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-rgw (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:09:01.654 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package libsnappy1v5:amd64. 2021-02-11T06:09:01.668 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../016-libsnappy1v5_1.1.7-1_amd64.deb ... 2021-02-11T06:09:01.704 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking libsnappy1v5:amd64 (1.1.7-1) ... 2021-02-11T06:09:02.015 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package libleveldb1v5:amd64. 2021-02-11T06:09:02.161 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../017-libleveldb1v5_1.20-2_amd64.deb ... 2021-02-11T06:09:02.161 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking libleveldb1v5:amd64 (1.20-2) ... 2021-02-11T06:09:02.452 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package liboath0. 2021-02-11T06:09:02.466 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../018-liboath0_2.6.1-1_amd64.deb ... 2021-02-11T06:09:02.510 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking liboath0 (2.6.1-1) ... 2021-02-11T06:09:02.813 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package libradosstriper1. 2021-02-11T06:09:02.826 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../019-libradosstriper1_17.0.0-711-g84a8025e-1bionic_amd64.deb ... 2021-02-11T06:09:02.871 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking libradosstriper1 (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:09:03.176 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-base. 2021-02-11T06:09:03.191 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../021-ceph-base_17.0.0-711-g84a8025e-1bionic_amd64.deb ... 2021-02-11T06:09:03.244 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-base (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:09:03.333 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package ceph-common. 2021-02-11T06:09:03.347 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../020-ceph-common_17.0.0-711-g84a8025e-1bionic_amd64.deb ... 2021-02-11T06:09:03.391 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking ceph-common (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:09:04.315 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-dateutil. 2021-02-11T06:09:04.328 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../022-python3-dateutil_2.6.1-1_all.deb ... 2021-02-11T06:09:04.373 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-dateutil (2.6.1-1) ... 2021-02-11T06:09:04.675 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-asn1crypto. 2021-02-11T06:09:04.689 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../023-python3-asn1crypto_0.24.0-1_all.deb ... 2021-02-11T06:09:04.725 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-asn1crypto (0.24.0-1) ... 2021-02-11T06:09:04.987 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-cffi-backend. 2021-02-11T06:09:04.999 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../024-python3-cffi-backend_1.11.5-1_amd64.deb ... 2021-02-11T06:09:05.036 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-cffi-backend (1.11.5-1) ... 2021-02-11T06:09:05.330 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package ceph-base. 2021-02-11T06:09:05.343 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../021-ceph-base_17.0.0-711-g84a8025e-1bionic_amd64.deb ... 2021-02-11T06:09:05.381 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-cryptography. 2021-02-11T06:09:05.389 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking ceph-base (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:09:05.395 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../025-python3-cryptography_2.1.4-1ubuntu1.4_amd64.deb ... 2021-02-11T06:09:05.439 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-cryptography (2.1.4-1ubuntu1.4) ... 2021-02-11T06:09:05.792 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-openssl. 2021-02-11T06:09:05.807 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../026-python3-openssl_17.5.0-1ubuntu1_all.deb ... 2021-02-11T06:09:05.850 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-openssl (17.5.0-1ubuntu1) ... 2021-02-11T06:09:06.120 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2021-02-11T06:09:06.135 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../027-ceph-mgr-modules-core_17.0.0-711-g84a8025e-1bionic_all.deb ... 2021-02-11T06:09:06.187 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-mgr-modules-core (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:09:06.326 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-dateutil. 2021-02-11T06:09:06.342 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../022-python3-dateutil_2.6.1-1_all.deb ... 2021-02-11T06:09:06.385 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-dateutil (2.6.1-1) ... 2021-02-11T06:09:06.558 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-bcrypt. 2021-02-11T06:09:06.571 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../028-python3-bcrypt_3.1.4-2_amd64.deb ... 2021-02-11T06:09:06.616 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-bcrypt (3.1.4-2) ... 2021-02-11T06:09:06.687 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-asn1crypto. 2021-02-11T06:09:06.703 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../023-python3-asn1crypto_0.24.0-1_all.deb ... 2021-02-11T06:09:06.745 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-asn1crypto (0.24.0-1) ... 2021-02-11T06:09:06.927 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-cherrypy3. 2021-02-11T06:09:06.940 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../029-python3-cherrypy3_8.9.1-2_all.deb ... 2021-02-11T06:09:06.976 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-cherrypy3 (8.9.1-2) ... 2021-02-11T06:09:07.032 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-cffi-backend. 2021-02-11T06:09:07.048 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../024-python3-cffi-backend_1.11.5-1_amd64.deb ... 2021-02-11T06:09:07.090 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-cffi-backend (1.11.5-1) ... 2021-02-11T06:09:07.330 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-jwt. 2021-02-11T06:09:07.345 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../030-python3-jwt_1.5.3+ds1-1_all.deb ... 2021-02-11T06:09:07.388 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-jwt (1.5.3+ds1-1) ... 2021-02-11T06:09:07.409 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-cryptography. 2021-02-11T06:09:07.423 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../025-python3-cryptography_2.1.4-1ubuntu1.4_amd64.deb ... 2021-02-11T06:09:07.460 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-cryptography (2.1.4-1ubuntu1.4) ... 2021-02-11T06:09:07.716 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-logutils. 2021-02-11T06:09:07.731 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../031-python3-logutils_0.3.3-5_all.deb ... 2021-02-11T06:09:07.783 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-logutils (0.3.3-5) ... 2021-02-11T06:09:07.921 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-openssl. 2021-02-11T06:09:07.936 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../026-python3-openssl_17.5.0-1ubuntu1_all.deb ... 2021-02-11T06:09:07.979 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-openssl (17.5.0-1ubuntu1) ... 2021-02-11T06:09:08.077 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-markupsafe. 2021-02-11T06:09:08.093 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../032-python3-markupsafe_1.0-1build1_amd64.deb ... 2021-02-11T06:09:08.135 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-markupsafe (1.0-1build1) ... 2021-02-11T06:09:08.265 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2021-02-11T06:09:08.280 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../027-ceph-mgr-modules-core_17.0.0-711-g84a8025e-1bionic_all.deb ... 2021-02-11T06:09:08.332 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking ceph-mgr-modules-core (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:09:08.453 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-mako. 2021-02-11T06:09:08.467 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../033-python3-mako_1.0.7+ds1-1_all.deb ... 2021-02-11T06:09:08.519 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-mako (1.0.7+ds1-1) ... 2021-02-11T06:09:08.712 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-bcrypt. 2021-02-11T06:09:08.726 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../028-python3-bcrypt_3.1.4-2_amd64.deb ... 2021-02-11T06:09:08.761 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-bcrypt (3.1.4-2) ... 2021-02-11T06:09:08.833 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-simplegeneric. 2021-02-11T06:09:08.849 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../034-python3-simplegeneric_0.8.1-1_all.deb ... 2021-02-11T06:09:08.891 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-simplegeneric (0.8.1-1) ... 2021-02-11T06:09:09.031 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-cherrypy3. 2021-02-11T06:09:09.045 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../029-python3-cherrypy3_8.9.1-2_all.deb ... 2021-02-11T06:09:09.081 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-cherrypy3 (8.9.1-2) ... 2021-02-11T06:09:17.776 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-singledispatch. 2021-02-11T06:09:17.777 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../035-python3-singledispatch_3.4.0.3-2_all.deb ... 2021-02-11T06:09:17.777 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-singledispatch (3.4.0.3-2) ... 2021-02-11T06:09:17.777 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-webob. 2021-02-11T06:09:17.777 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../036-python3-webob_1%3a1.7.3-2fakesync1_all.deb ... 2021-02-11T06:09:17.777 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-webob (1:1.7.3-2fakesync1) ... 2021-02-11T06:09:17.777 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-bs4. 2021-02-11T06:09:17.777 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../037-python3-bs4_4.6.0-1_all.deb ... 2021-02-11T06:09:17.777 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-bs4 (4.6.0-1) ... 2021-02-11T06:09:17.777 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-waitress. 2021-02-11T06:09:17.777 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../038-python3-waitress_1.0.1-1_all.deb ... 2021-02-11T06:09:17.778 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-waitress (1.0.1-1) ... 2021-02-11T06:09:17.778 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-tempita. 2021-02-11T06:09:17.778 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../039-python3-tempita_0.5.2-2_all.deb ... 2021-02-11T06:09:17.778 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-tempita (0.5.2-2) ... 2021-02-11T06:09:17.778 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-paste. 2021-02-11T06:09:17.778 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../040-python3-paste_2.0.3+dfsg-4ubuntu1_all.deb ... 2021-02-11T06:09:17.778 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-paste (2.0.3+dfsg-4ubuntu1) ... 2021-02-11T06:09:17.778 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2021-02-11T06:09:17.778 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../041-python-pastedeploy-tpl_1.5.2-4_all.deb ... 2021-02-11T06:09:17.778 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python-pastedeploy-tpl (1.5.2-4) ... 2021-02-11T06:09:17.778 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-pastedeploy. 2021-02-11T06:09:17.779 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../042-python3-pastedeploy_1.5.2-4_all.deb ... 2021-02-11T06:09:17.779 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-pastedeploy (1.5.2-4) ... 2021-02-11T06:09:17.779 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-webtest. 2021-02-11T06:09:17.779 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../043-python3-webtest_2.0.28-1ubuntu1_all.deb ... 2021-02-11T06:09:17.779 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-webtest (2.0.28-1ubuntu1) ... 2021-02-11T06:09:17.779 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-pecan. 2021-02-11T06:09:17.779 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../044-python3-pecan_1.2.1-2_all.deb ... 2021-02-11T06:09:17.779 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-pecan (1.2.1-2) ... 2021-02-11T06:09:17.779 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-werkzeug. 2021-02-11T06:09:17.779 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../045-python3-werkzeug_0.14.1+dfsg1-1ubuntu0.1_all.deb ... 2021-02-11T06:09:17.779 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-werkzeug (0.14.1+dfsg1-1ubuntu0.1) ... 2021-02-11T06:09:17.779 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-mgr. 2021-02-11T06:09:17.779 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../046-ceph-mgr_17.0.0-711-g84a8025e-1bionic_amd64.deb ... 2021-02-11T06:09:17.779 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-mgr (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:09:17.779 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-mon. 2021-02-11T06:09:17.780 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../047-ceph-mon_17.0.0-711-g84a8025e-1bionic_amd64.deb ... 2021-02-11T06:09:17.780 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-mon (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:09:17.780 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-osd. 2021-02-11T06:09:17.780 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../048-ceph-osd_17.0.0-711-g84a8025e-1bionic_amd64.deb ... 2021-02-11T06:09:17.780 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-osd (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:09:17.780 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph. 2021-02-11T06:09:17.780 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../049-ceph_17.0.0-711-g84a8025e-1bionic_amd64.deb ... 2021-02-11T06:09:17.780 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:09:17.780 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-fuse. 2021-02-11T06:09:17.780 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../050-ceph-fuse_17.0.0-711-g84a8025e-1bionic_amd64.deb ... 2021-02-11T06:09:17.780 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-fuse (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:09:17.780 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-jwt. 2021-02-11T06:09:17.780 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../030-python3-jwt_1.5.3+ds1-1_all.deb ... 2021-02-11T06:09:17.781 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-jwt (1.5.3+ds1-1) ... 2021-02-11T06:09:17.781 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-logutils. 2021-02-11T06:09:17.781 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../031-python3-logutils_0.3.3-5_all.deb ... 2021-02-11T06:09:17.781 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-logutils (0.3.3-5) ... 2021-02-11T06:09:17.781 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-markupsafe. 2021-02-11T06:09:17.781 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../032-python3-markupsafe_1.0-1build1_amd64.deb ... 2021-02-11T06:09:17.781 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-markupsafe (1.0-1build1) ... 2021-02-11T06:09:17.781 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-mako. 2021-02-11T06:09:17.781 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../033-python3-mako_1.0.7+ds1-1_all.deb ... 2021-02-11T06:09:17.781 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-mako (1.0.7+ds1-1) ... 2021-02-11T06:09:17.781 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-simplegeneric. 2021-02-11T06:09:17.781 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../034-python3-simplegeneric_0.8.1-1_all.deb ... 2021-02-11T06:09:17.781 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-simplegeneric (0.8.1-1) ... 2021-02-11T06:09:17.781 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-singledispatch. 2021-02-11T06:09:17.782 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../035-python3-singledispatch_3.4.0.3-2_all.deb ... 2021-02-11T06:09:17.782 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-singledispatch (3.4.0.3-2) ... 2021-02-11T06:09:17.782 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-webob. 2021-02-11T06:09:17.782 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../036-python3-webob_1%3a1.7.3-2fakesync1_all.deb ... 2021-02-11T06:09:17.782 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-webob (1:1.7.3-2fakesync1) ... 2021-02-11T06:09:17.782 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-bs4. 2021-02-11T06:09:17.782 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../037-python3-bs4_4.6.0-1_all.deb ... 2021-02-11T06:09:17.782 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-bs4 (4.6.0-1) ... 2021-02-11T06:09:17.782 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-waitress. 2021-02-11T06:09:17.782 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../038-python3-waitress_1.0.1-1_all.deb ... 2021-02-11T06:09:17.782 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-waitress (1.0.1-1) ... 2021-02-11T06:09:17.782 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-tempita. 2021-02-11T06:09:17.782 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../039-python3-tempita_0.5.2-2_all.deb ... 2021-02-11T06:09:17.783 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-tempita (0.5.2-2) ... 2021-02-11T06:09:17.783 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-paste. 2021-02-11T06:09:17.783 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../040-python3-paste_2.0.3+dfsg-4ubuntu1_all.deb ... 2021-02-11T06:09:17.783 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-paste (2.0.3+dfsg-4ubuntu1) ... 2021-02-11T06:09:17.783 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2021-02-11T06:09:17.783 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../041-python-pastedeploy-tpl_1.5.2-4_all.deb ... 2021-02-11T06:09:17.783 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python-pastedeploy-tpl (1.5.2-4) ... 2021-02-11T06:09:17.783 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-pastedeploy. 2021-02-11T06:09:17.783 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../042-python3-pastedeploy_1.5.2-4_all.deb ... 2021-02-11T06:09:17.783 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-pastedeploy (1.5.2-4) ... 2021-02-11T06:09:17.783 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-webtest. 2021-02-11T06:09:17.783 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../043-python3-webtest_2.0.28-1ubuntu1_all.deb ... 2021-02-11T06:09:17.783 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-webtest (2.0.28-1ubuntu1) ... 2021-02-11T06:09:17.783 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-pecan. 2021-02-11T06:09:17.784 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../044-python3-pecan_1.2.1-2_all.deb ... 2021-02-11T06:09:17.784 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-pecan (1.2.1-2) ... 2021-02-11T06:09:17.784 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-werkzeug. 2021-02-11T06:09:17.784 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../045-python3-werkzeug_0.14.1+dfsg1-1ubuntu0.1_all.deb ... 2021-02-11T06:09:17.784 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-werkzeug (0.14.1+dfsg1-1ubuntu0.1) ... 2021-02-11T06:09:17.784 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package ceph-mgr. 2021-02-11T06:09:17.784 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../046-ceph-mgr_17.0.0-711-g84a8025e-1bionic_amd64.deb ... 2021-02-11T06:09:17.784 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking ceph-mgr (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:09:17.784 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package ceph-mon. 2021-02-11T06:09:17.784 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../047-ceph-mon_17.0.0-711-g84a8025e-1bionic_amd64.deb ... 2021-02-11T06:09:17.784 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking ceph-mon (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:09:17.784 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package ceph-osd. 2021-02-11T06:09:17.784 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../048-ceph-osd_17.0.0-711-g84a8025e-1bionic_amd64.deb ... 2021-02-11T06:09:17.784 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking ceph-osd (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:09:18.017 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-mds. 2021-02-11T06:09:18.645 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../051-ceph-mds_17.0.0-711-g84a8025e-1bionic_amd64.deb ... 2021-02-11T06:09:18.645 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-mds (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:09:18.645 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package cephadm. 2021-02-11T06:09:18.645 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../052-cephadm_17.0.0-711-g84a8025e-1bionic_amd64.deb ... 2021-02-11T06:09:18.656 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking cephadm (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:09:18.840 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package ceph. 2021-02-11T06:09:18.856 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../049-ceph_17.0.0-711-g84a8025e-1bionic_amd64.deb ... 2021-02-11T06:09:18.890 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking ceph (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:09:19.001 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-jinja2. 2021-02-11T06:09:19.016 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../053-python3-jinja2_2.10-1ubuntu0.18.04.1_all.deb ... 2021-02-11T06:09:19.059 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2021-02-11T06:09:19.210 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package ceph-fuse. 2021-02-11T06:09:19.225 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../050-ceph-fuse_17.0.0-711-g84a8025e-1bionic_amd64.deb ... 2021-02-11T06:09:19.268 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking ceph-fuse (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:09:19.388 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2021-02-11T06:09:19.404 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../054-ceph-mgr-cephadm_17.0.0-711-g84a8025e-1bionic_all.deb ... 2021-02-11T06:09:19.455 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-mgr-cephadm (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:09:19.688 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package ceph-mds. 2021-02-11T06:09:19.702 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../051-ceph-mds_17.0.0-711-g84a8025e-1bionic_amd64.deb ... 2021-02-11T06:09:19.746 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking ceph-mds (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:09:19.784 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-repoze.lru. 2021-02-11T06:09:19.798 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../055-python3-repoze.lru_0.7-1_all.deb ... 2021-02-11T06:09:19.841 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-repoze.lru (0.7-1) ... 2021-02-11T06:09:20.136 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-routes. 2021-02-11T06:09:20.152 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../056-python3-routes_2.4.1-1_all.deb ... 2021-02-11T06:09:20.203 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-routes (2.4.1-1) ... 2021-02-11T06:09:20.250 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package cephadm. 2021-02-11T06:09:20.265 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../052-cephadm_17.0.0-711-g84a8025e-1bionic_amd64.deb ... 2021-02-11T06:09:20.308 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking cephadm (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:09:20.527 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2021-02-11T06:09:20.544 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../057-ceph-mgr-dashboard_17.0.0-711-g84a8025e-1bionic_all.deb ... 2021-02-11T06:09:20.586 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-mgr-dashboard (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:09:20.611 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-jinja2. 2021-02-11T06:09:20.627 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../053-python3-jinja2_2.10-1ubuntu0.18.04.1_all.deb ... 2021-02-11T06:09:20.669 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2021-02-11T06:09:20.982 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2021-02-11T06:09:20.996 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../054-ceph-mgr-cephadm_17.0.0-711-g84a8025e-1bionic_all.deb ... 2021-02-11T06:09:21.032 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking ceph-mgr-cephadm (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:09:21.369 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-repoze.lru. 2021-02-11T06:09:21.383 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../055-python3-repoze.lru_0.7-1_all.deb ... 2021-02-11T06:09:21.428 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-numpy. 2021-02-11T06:09:21.428 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-repoze.lru (0.7-1) ... 2021-02-11T06:09:21.445 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../058-python3-numpy_1%3a1.13.3-2ubuntu1_amd64.deb ... 2021-02-11T06:09:21.486 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-numpy (1:1.13.3-2ubuntu1) ... 2021-02-11T06:09:21.715 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-routes. 2021-02-11T06:09:21.731 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../056-python3-routes_2.4.1-1_all.deb ... 2021-02-11T06:09:21.773 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-routes (2.4.1-1) ... 2021-02-11T06:09:22.059 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-decorator. 2021-02-11T06:09:22.075 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../059-python3-decorator_4.1.2-1_all.deb ... 2021-02-11T06:09:22.101 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2021-02-11T06:09:22.117 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../057-ceph-mgr-dashboard_17.0.0-711-g84a8025e-1bionic_all.deb ... 2021-02-11T06:09:22.117 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-decorator (4.1.2-1) ... 2021-02-11T06:09:22.160 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking ceph-mgr-dashboard (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:09:22.453 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-scipy. 2021-02-11T06:09:22.469 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../060-python3-scipy_0.19.1-2ubuntu1_amd64.deb ... 2021-02-11T06:09:22.511 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-scipy (0.19.1-2ubuntu1) ... 2021-02-11T06:09:23.040 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-numpy. 2021-02-11T06:09:23.055 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../058-python3-numpy_1%3a1.13.3-2ubuntu1_amd64.deb ... 2021-02-11T06:09:23.098 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-numpy (1:1.13.3-2ubuntu1) ... 2021-02-11T06:09:23.637 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-decorator. 2021-02-11T06:09:23.651 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../059-python3-decorator_4.1.2-1_all.deb ... 2021-02-11T06:09:23.687 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-decorator (4.1.2-1) ... 2021-02-11T06:09:23.861 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-sklearn-lib. 2021-02-11T06:09:23.876 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../061-python3-sklearn-lib_0.19.1-3_amd64.deb ... 2021-02-11T06:09:23.928 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-sklearn-lib (0.19.1-3) ... 2021-02-11T06:09:23.948 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-scipy. 2021-02-11T06:09:23.964 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../060-python3-scipy_0.19.1-2ubuntu1_amd64.deb ... 2021-02-11T06:09:23.998 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-scipy (0.19.1-2ubuntu1) ... 2021-02-11T06:09:24.423 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-joblib. 2021-02-11T06:09:37.801 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-sklearn-lib. 2021-02-11T06:09:37.801 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../061-python3-sklearn-lib_0.19.1-3_amd64.deb ... 2021-02-11T06:09:37.802 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-sklearn-lib (0.19.1-3) ... 2021-02-11T06:09:37.802 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-joblib. 2021-02-11T06:09:37.802 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../062-python3-joblib_0.11-1_all.deb ... 2021-02-11T06:09:37.802 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-joblib (0.11-1) ... 2021-02-11T06:09:37.802 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-sklearn. 2021-02-11T06:09:37.802 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../063-python3-sklearn_0.19.1-3_all.deb ... 2021-02-11T06:09:37.802 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-sklearn (0.19.1-3) ... 2021-02-11T06:09:37.802 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2021-02-11T06:09:37.802 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../064-ceph-mgr-diskprediction-local_17.0.0-711-g84a8025e-1bionic_all.deb ... 2021-02-11T06:09:37.802 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking ceph-mgr-diskprediction-local (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:09:37.802 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-monotonic. 2021-02-11T06:09:37.802 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../065-python3-monotonic_1.1-2_all.deb ... 2021-02-11T06:09:37.802 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-monotonic (1.1-2) ... 2021-02-11T06:09:37.803 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-fasteners. 2021-02-11T06:09:37.803 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../066-python3-fasteners_0.12.0-3_all.deb ... 2021-02-11T06:09:37.803 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-fasteners (0.12.0-3) ... 2021-02-11T06:09:37.803 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-httplib2. 2021-02-11T06:09:37.803 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../067-python3-httplib2_0.9.2+dfsg-1ubuntu0.2_all.deb ... 2021-02-11T06:09:37.803 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2021-02-11T06:09:37.803 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-pyasn1. 2021-02-11T06:09:37.803 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../068-python3-pyasn1_0.4.2-3_all.deb ... 2021-02-11T06:09:37.803 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-pyasn1 (0.4.2-3) ... 2021-02-11T06:09:37.803 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-pyasn1-modules. 2021-02-11T06:09:37.803 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../069-python3-pyasn1-modules_0.2.1-0.2_all.deb ... 2021-02-11T06:09:37.803 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2) ... 2021-02-11T06:09:37.803 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-rsa. 2021-02-11T06:09:37.804 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../070-python3-rsa_3.4.2-1_all.deb ... 2021-02-11T06:09:37.804 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-rsa (3.4.2-1) ... 2021-02-11T06:09:37.804 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-sqlalchemy. 2021-02-11T06:09:37.804 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../071-python3-sqlalchemy_1.1.11+ds1-1ubuntu1_all.deb ... 2021-02-11T06:09:37.804 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2021-02-11T06:09:37.804 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-oauth2client. 2021-02-11T06:09:37.804 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../072-python3-oauth2client_4.1.2-3_all.deb ... 2021-02-11T06:09:37.804 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-oauth2client (4.1.2-3) ... 2021-02-11T06:09:37.804 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-setuptools. 2021-02-11T06:09:37.804 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../073-python3-setuptools_39.0.1-2_all.deb ... 2021-02-11T06:09:37.804 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-setuptools (39.0.1-2) ... 2021-02-11T06:09:37.804 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-websocket. 2021-02-11T06:09:37.804 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../074-python3-websocket_0.44.0-0ubuntu2_all.deb ... 2021-02-11T06:09:37.804 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-websocket (0.44.0-0ubuntu2) ... 2021-02-11T06:09:37.804 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-kubernetes. 2021-02-11T06:09:37.805 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../075-python3-kubernetes_2.0.0-2ubuntu1_all.deb ... 2021-02-11T06:09:37.805 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-kubernetes (2.0.0-2ubuntu1) ... 2021-02-11T06:09:37.805 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2021-02-11T06:09:37.805 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../076-ceph-mgr-k8sevents_17.0.0-711-g84a8025e-1bionic_all.deb ... 2021-02-11T06:09:37.805 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking ceph-mgr-k8sevents (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:09:37.805 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-json-pointer. 2021-02-11T06:09:37.805 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../077-python3-json-pointer_1.10-1_all.deb ... 2021-02-11T06:09:37.805 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-json-pointer (1.10-1) ... 2021-02-11T06:09:37.805 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-jsonpatch. 2021-02-11T06:09:37.805 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../078-python3-jsonpatch_1.19+really1.16-1fakesync1_all.deb ... 2021-02-11T06:09:37.805 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2021-02-11T06:09:37.805 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package ceph-mgr-rook. 2021-02-11T06:09:37.805 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../079-ceph-mgr-rook_17.0.0-711-g84a8025e-1bionic_all.deb ... 2021-02-11T06:09:37.805 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking ceph-mgr-rook (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:09:37.806 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package libonig4:amd64. 2021-02-11T06:09:37.806 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../080-libonig4_6.7.0-1_amd64.deb ... 2021-02-11T06:09:37.806 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking libonig4:amd64 (6.7.0-1) ... 2021-02-11T06:09:37.806 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package libjq1:amd64. 2021-02-11T06:09:37.806 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../081-libjq1_1.5+dfsg-2_amd64.deb ... 2021-02-11T06:09:37.806 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking libjq1:amd64 (1.5+dfsg-2) ... 2021-02-11T06:09:37.806 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package jq. 2021-02-11T06:09:37.806 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../082-jq_1.5+dfsg-2_amd64.deb ... 2021-02-11T06:09:37.806 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking jq (1.5+dfsg-2) ... 2021-02-11T06:09:37.806 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package socat. 2021-02-11T06:09:37.806 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../083-socat_1.7.3.2-2ubuntu2_amd64.deb ... 2021-02-11T06:09:37.806 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking socat (1.7.3.2-2ubuntu2) ... 2021-02-11T06:09:37.806 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package libxslt1.1:amd64. 2021-02-11T06:09:37.807 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../084-libxslt1.1_1.1.29-5ubuntu0.2_amd64.deb ... 2021-02-11T06:09:37.807 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2021-02-11T06:09:37.807 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package xmlstarlet. 2021-02-11T06:09:37.807 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../085-xmlstarlet_1.6.1-2_amd64.deb ... 2021-02-11T06:09:37.807 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking xmlstarlet (1.6.1-2) ... 2021-02-11T06:09:37.807 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package ceph-test. 2021-02-11T06:09:37.807 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../086-ceph-test_17.0.0-711-g84a8025e-1bionic_amd64.deb ... 2021-02-11T06:09:37.807 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking ceph-test (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:09:37.807 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../062-python3-joblib_0.11-1_all.deb ... 2021-02-11T06:09:37.807 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-joblib (0.11-1) ... 2021-02-11T06:09:37.807 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-sklearn. 2021-02-11T06:09:37.807 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../063-python3-sklearn_0.19.1-3_all.deb ... 2021-02-11T06:09:37.807 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-sklearn (0.19.1-3) ... 2021-02-11T06:09:37.807 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2021-02-11T06:09:37.808 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../064-ceph-mgr-diskprediction-local_17.0.0-711-g84a8025e-1bionic_all.deb ... 2021-02-11T06:09:37.808 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-mgr-diskprediction-local (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:09:37.808 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-monotonic. 2021-02-11T06:09:37.808 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../065-python3-monotonic_1.1-2_all.deb ... 2021-02-11T06:09:37.808 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-monotonic (1.1-2) ... 2021-02-11T06:09:37.808 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-fasteners. 2021-02-11T06:09:37.808 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../066-python3-fasteners_0.12.0-3_all.deb ... 2021-02-11T06:09:37.808 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-fasteners (0.12.0-3) ... 2021-02-11T06:09:37.808 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-httplib2. 2021-02-11T06:09:37.808 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../067-python3-httplib2_0.9.2+dfsg-1ubuntu0.2_all.deb ... 2021-02-11T06:09:37.808 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2021-02-11T06:09:37.808 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-pyasn1. 2021-02-11T06:09:37.808 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../068-python3-pyasn1_0.4.2-3_all.deb ... 2021-02-11T06:09:37.808 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-pyasn1 (0.4.2-3) ... 2021-02-11T06:09:37.809 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-pyasn1-modules. 2021-02-11T06:09:37.809 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../069-python3-pyasn1-modules_0.2.1-0.2_all.deb ... 2021-02-11T06:09:37.809 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2) ... 2021-02-11T06:09:37.809 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-rsa. 2021-02-11T06:09:37.809 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../070-python3-rsa_3.4.2-1_all.deb ... 2021-02-11T06:09:37.809 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-rsa (3.4.2-1) ... 2021-02-11T06:09:37.809 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-sqlalchemy. 2021-02-11T06:09:37.809 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../071-python3-sqlalchemy_1.1.11+ds1-1ubuntu1_all.deb ... 2021-02-11T06:09:37.809 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2021-02-11T06:09:37.809 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-oauth2client. 2021-02-11T06:09:37.809 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../072-python3-oauth2client_4.1.2-3_all.deb ... 2021-02-11T06:09:37.809 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-oauth2client (4.1.2-3) ... 2021-02-11T06:09:37.809 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-setuptools. 2021-02-11T06:09:37.809 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../073-python3-setuptools_39.0.1-2_all.deb ... 2021-02-11T06:09:37.809 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-setuptools (39.0.1-2) ... 2021-02-11T06:09:37.810 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-websocket. 2021-02-11T06:09:37.810 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../074-python3-websocket_0.44.0-0ubuntu2_all.deb ... 2021-02-11T06:09:37.810 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-websocket (0.44.0-0ubuntu2) ... 2021-02-11T06:09:37.810 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-kubernetes. 2021-02-11T06:09:37.810 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../075-python3-kubernetes_2.0.0-2ubuntu1_all.deb ... 2021-02-11T06:09:37.810 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-kubernetes (2.0.0-2ubuntu1) ... 2021-02-11T06:09:37.810 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2021-02-11T06:09:37.810 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../076-ceph-mgr-k8sevents_17.0.0-711-g84a8025e-1bionic_all.deb ... 2021-02-11T06:09:37.810 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-mgr-k8sevents (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:09:37.810 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-json-pointer. 2021-02-11T06:09:37.810 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../077-python3-json-pointer_1.10-1_all.deb ... 2021-02-11T06:09:37.810 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-json-pointer (1.10-1) ... 2021-02-11T06:09:37.811 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-jsonpatch. 2021-02-11T06:09:37.811 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../078-python3-jsonpatch_1.19+really1.16-1fakesync1_all.deb ... 2021-02-11T06:09:37.811 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2021-02-11T06:09:37.811 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-mgr-rook. 2021-02-11T06:09:37.811 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../079-ceph-mgr-rook_17.0.0-711-g84a8025e-1bionic_all.deb ... 2021-02-11T06:09:37.811 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-mgr-rook (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:09:37.811 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libonig4:amd64. 2021-02-11T06:09:37.811 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../080-libonig4_6.7.0-1_amd64.deb ... 2021-02-11T06:09:37.811 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libonig4:amd64 (6.7.0-1) ... 2021-02-11T06:09:37.811 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libjq1:amd64. 2021-02-11T06:09:37.811 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../081-libjq1_1.5+dfsg-2_amd64.deb ... 2021-02-11T06:09:37.811 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libjq1:amd64 (1.5+dfsg-2) ... 2021-02-11T06:09:37.811 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package jq. 2021-02-11T06:09:37.811 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../082-jq_1.5+dfsg-2_amd64.deb ... 2021-02-11T06:09:37.811 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking jq (1.5+dfsg-2) ... 2021-02-11T06:09:37.812 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package socat. 2021-02-11T06:09:37.812 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../083-socat_1.7.3.2-2ubuntu2_amd64.deb ... 2021-02-11T06:09:37.812 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking socat (1.7.3.2-2ubuntu2) ... 2021-02-11T06:09:37.812 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libxslt1.1:amd64. 2021-02-11T06:09:37.812 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../084-libxslt1.1_1.1.29-5ubuntu0.2_amd64.deb ... 2021-02-11T06:09:37.812 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2021-02-11T06:09:37.812 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package xmlstarlet. 2021-02-11T06:09:37.812 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../085-xmlstarlet_1.6.1-2_amd64.deb ... 2021-02-11T06:09:37.812 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking xmlstarlet (1.6.1-2) ... 2021-02-11T06:09:37.812 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-test. 2021-02-11T06:09:37.812 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../086-ceph-test_17.0.0-711-g84a8025e-1bionic_amd64.deb ... 2021-02-11T06:09:37.812 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-test (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:09:39.210 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libcephfs-dev. 2021-02-11T06:09:39.227 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../087-libcephfs-dev_17.0.0-711-g84a8025e-1bionic_amd64.deb ... 2021-02-11T06:09:39.285 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libcephfs-dev (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:09:39.571 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package nvme-cli. 2021-02-11T06:09:39.587 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../088-nvme-cli_1.5-1ubuntu1.1_amd64.deb ... 2021-02-11T06:09:39.629 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking nvme-cli (1.5-1ubuntu1.1) ... 2021-02-11T06:09:39.968 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python-attr. 2021-02-11T06:09:39.982 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../089-python-attr_17.4.0-2_all.deb ... 2021-02-11T06:09:39.989 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package libcephfs-dev. 2021-02-11T06:09:40.006 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../087-libcephfs-dev_17.0.0-711-g84a8025e-1bionic_amd64.deb ... 2021-02-11T06:09:40.026 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python-attr (17.4.0-2) ... 2021-02-11T06:09:40.039 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking libcephfs-dev (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:09:40.317 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package nvme-cli. 2021-02-11T06:09:40.329 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python-funcsigs. 2021-02-11T06:09:40.331 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../088-nvme-cli_1.5-1ubuntu1.1_amd64.deb ... 2021-02-11T06:09:40.343 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../090-python-funcsigs_1.0.2-4_all.deb ... 2021-02-11T06:09:40.375 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking nvme-cli (1.5-1ubuntu1.1) ... 2021-02-11T06:09:40.387 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python-funcsigs (1.0.2-4) ... 2021-02-11T06:09:40.713 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python-attr. 2021-02-11T06:09:40.715 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python-pluggy. 2021-02-11T06:09:40.727 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../089-python-attr_17.4.0-2_all.deb ... 2021-02-11T06:09:40.732 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../091-python-pluggy_0.6.0-1_all.deb ... 2021-02-11T06:09:40.763 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python-attr (17.4.0-2) ... 2021-02-11T06:09:40.795 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python-pluggy (0.6.0-1) ... 2021-02-11T06:09:41.032 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python-funcsigs. 2021-02-11T06:09:41.046 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../090-python-funcsigs_1.0.2-4_all.deb ... 2021-02-11T06:09:41.082 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python-funcsigs (1.0.2-4) ... 2021-02-11T06:09:41.101 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python-py. 2021-02-11T06:09:41.118 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../092-python-py_1.5.2-1_all.deb ... 2021-02-11T06:09:41.160 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python-py (1.5.2-1) ... 2021-02-11T06:09:41.411 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python-pluggy. 2021-02-11T06:09:41.427 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../091-python-pluggy_0.6.0-1_all.deb ... 2021-02-11T06:09:41.469 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python-pluggy (0.6.0-1) ... 2021-02-11T06:09:41.504 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python-pytest. 2021-02-11T06:09:41.521 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../093-python-pytest_3.3.2-2_all.deb ... 2021-02-11T06:09:41.562 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python-pytest (3.3.2-2) ... 2021-02-11T06:09:41.772 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python-py. 2021-02-11T06:09:41.787 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../092-python-py_1.5.2-1_all.deb ... 2021-02-11T06:09:41.830 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python-py (1.5.2-1) ... 2021-02-11T06:09:41.941 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-attr. 2021-02-11T06:09:41.957 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../094-python3-attr_17.4.0-2_all.deb ... 2021-02-11T06:09:42.008 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-attr (17.4.0-2) ... 2021-02-11T06:09:42.142 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python-pytest. 2021-02-11T06:09:42.157 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../093-python-pytest_3.3.2-2_all.deb ... 2021-02-11T06:09:42.200 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python-pytest (3.3.2-2) ... 2021-02-11T06:09:42.352 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-cycler. 2021-02-11T06:09:42.366 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../095-python3-cycler_0.10.0-1_all.deb ... 2021-02-11T06:09:42.410 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-cycler (0.10.0-1) ... 2021-02-11T06:09:42.537 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-attr. 2021-02-11T06:09:42.551 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../094-python3-attr_17.4.0-2_all.deb ... 2021-02-11T06:09:42.595 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-attr (17.4.0-2) ... 2021-02-11T06:09:42.713 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-webencodings. 2021-02-11T06:09:42.729 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../096-python3-webencodings_0.5-2_all.deb ... 2021-02-11T06:09:42.771 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-webencodings (0.5-2) ... 2021-02-11T06:09:42.907 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-cycler. 2021-02-11T06:09:42.922 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../095-python3-cycler_0.10.0-1_all.deb ... 2021-02-11T06:09:42.965 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-cycler (0.10.0-1) ... 2021-02-11T06:09:43.075 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-html5lib. 2021-02-11T06:09:43.091 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../097-python3-html5lib_0.999999999-1_all.deb ... 2021-02-11T06:09:43.141 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-html5lib (0.999999999-1) ... 2021-02-11T06:09:43.276 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-webencodings. 2021-02-11T06:09:43.290 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../096-python3-webencodings_0.5-2_all.deb ... 2021-02-11T06:09:43.334 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-webencodings (0.5-2) ... 2021-02-11T06:09:43.503 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-lxml:amd64. 2021-02-11T06:09:43.519 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../098-python3-lxml_4.2.1-1ubuntu0.3_amd64.deb ... 2021-02-11T06:09:43.561 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-lxml:amd64 (4.2.1-1ubuntu0.3) ... 2021-02-11T06:09:43.637 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-html5lib. 2021-02-11T06:09:43.651 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../097-python3-html5lib_0.999999999-1_all.deb ... 2021-02-11T06:09:43.695 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-html5lib (0.999999999-1) ... 2021-02-11T06:09:44.049 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-lxml:amd64. 2021-02-11T06:09:44.062 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../098-python3-lxml_4.2.1-1ubuntu0.3_amd64.deb ... 2021-02-11T06:09:44.064 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-pyparsing. 2021-02-11T06:09:44.080 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../099-python3-pyparsing_2.2.0+dfsg1-2_all.deb ... 2021-02-11T06:09:44.107 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-lxml:amd64 (4.2.1-1ubuntu0.3) ... 2021-02-11T06:09:44.130 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-pyparsing (2.2.0+dfsg1-2) ... 2021-02-11T06:09:44.450 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-tz. 2021-02-11T06:09:44.467 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../100-python3-tz_2018.3-2_all.deb ... 2021-02-11T06:09:44.508 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-tz (2018.3-2) ... 2021-02-11T06:09:44.560 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-pyparsing. 2021-02-11T06:09:44.576 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../099-python3-pyparsing_2.2.0+dfsg1-2_all.deb ... 2021-02-11T06:09:44.627 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-pyparsing (2.2.0+dfsg1-2) ... 2021-02-11T06:09:44.812 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-matplotlib. 2021-02-11T06:09:44.828 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../101-python3-matplotlib_2.1.1-2ubuntu3_amd64.deb ... 2021-02-11T06:09:44.870 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-matplotlib (2.1.1-2ubuntu3) ... 2021-02-11T06:09:44.946 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-tz. 2021-02-11T06:09:44.962 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../100-python3-tz_2018.3-2_all.deb ... 2021-02-11T06:09:45.004 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-tz (2018.3-2) ... 2021-02-11T06:09:45.307 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-matplotlib. 2021-02-11T06:09:45.322 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../101-python3-matplotlib_2.1.1-2ubuntu3_amd64.deb ... 2021-02-11T06:09:45.365 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-matplotlib (2.1.1-2ubuntu3) ... 2021-02-11T06:09:45.536 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-olefile. 2021-02-11T06:09:45.549 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../102-python3-olefile_0.45.1-1_all.deb ... 2021-02-11T06:09:45.594 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-olefile (0.45.1-1) ... 2021-02-11T06:09:45.897 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-pastescript. 2021-02-11T06:09:45.911 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../103-python3-pastescript_2.0.2-2_all.deb ... 2021-02-11T06:09:45.955 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-pastescript (2.0.2-2) ... 2021-02-11T06:09:46.021 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-olefile. 2021-02-11T06:09:46.037 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../102-python3-olefile_0.45.1-1_all.deb ... 2021-02-11T06:09:46.079 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-olefile (0.45.1-1) ... 2021-02-11T06:09:46.326 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-pil:amd64. 2021-02-11T06:09:46.341 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../104-python3-pil_5.1.0-1ubuntu0.4_amd64.deb ... 2021-02-11T06:09:46.384 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-pil:amd64 (5.1.0-1ubuntu0.4) ... 2021-02-11T06:09:46.408 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-pastescript. 2021-02-11T06:09:46.424 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../103-python3-pastescript_2.0.2-2_all.deb ... 2021-02-11T06:09:46.474 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-pastescript (2.0.2-2) ... 2021-02-11T06:09:46.745 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-pluggy. 2021-02-11T06:09:46.761 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../105-python3-pluggy_0.6.0-1_all.deb ... 2021-02-11T06:09:46.803 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-pluggy (0.6.0-1) ... 2021-02-11T06:09:46.887 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-pil:amd64. 2021-02-11T06:09:46.901 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../104-python3-pil_5.1.0-1ubuntu0.4_amd64.deb ... 2021-02-11T06:09:46.944 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-pil:amd64 (5.1.0-1ubuntu0.4) ... 2021-02-11T06:09:47.123 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-py. 2021-02-11T06:09:47.140 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../106-python3-py_1.5.2-1_all.deb ... 2021-02-11T06:09:47.181 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-py (1.5.2-1) ... 2021-02-11T06:09:47.306 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-pluggy. 2021-02-11T06:09:47.323 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../105-python3-pluggy_0.6.0-1_all.deb ... 2021-02-11T06:09:47.364 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-pluggy (0.6.0-1) ... 2021-02-11T06:09:47.518 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-pyinotify. 2021-02-11T06:09:47.535 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../107-python3-pyinotify_0.9.6-1_all.deb ... 2021-02-11T06:09:47.576 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-pyinotify (0.9.6-1) ... 2021-02-11T06:09:47.692 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-py. 2021-02-11T06:09:47.706 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../106-python3-py_1.5.2-1_all.deb ... 2021-02-11T06:09:47.750 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-py (1.5.2-1) ... 2021-02-11T06:09:47.879 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-pytest. 2021-02-11T06:09:47.896 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../108-python3-pytest_3.3.2-2_all.deb ... 2021-02-11T06:09:47.937 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-pytest (3.3.2-2) ... 2021-02-11T06:09:48.087 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-pyinotify. 2021-02-11T06:09:48.102 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../107-python3-pyinotify_0.9.6-1_all.deb ... 2021-02-11T06:09:48.145 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-pyinotify (0.9.6-1) ... 2021-02-11T06:09:48.276 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-simplejson. 2021-02-11T06:09:48.291 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../109-python3-simplejson_3.13.2-1_amd64.deb ... 2021-02-11T06:09:48.334 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-simplejson (3.13.2-1) ... 2021-02-11T06:09:48.465 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-pytest. 2021-02-11T06:09:48.479 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../108-python3-pytest_3.3.2-2_all.deb ... 2021-02-11T06:09:48.523 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-pytest (3.3.2-2) ... 2021-02-11T06:09:48.687 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-tk:amd64. 2021-02-11T06:09:48.701 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../110-python3-tk_3.6.9-1~18.04_amd64.deb ... 2021-02-11T06:09:48.745 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-tk:amd64 (3.6.9-1~18.04) ... 2021-02-11T06:09:48.868 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-simplejson. 2021-02-11T06:09:48.883 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../109-python3-simplejson_3.13.2-1_amd64.deb ... 2021-02-11T06:09:48.952 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-simplejson (3.13.2-1) ... 2021-02-11T06:09:49.208 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package radosgw. 2021-02-11T06:09:57.080 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-tk:amd64. 2021-02-11T06:09:57.080 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../110-python3-tk_3.6.9-1~18.04_amd64.deb ... 2021-02-11T06:09:57.080 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-tk:amd64 (3.6.9-1~18.04) ... 2021-02-11T06:09:57.080 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package radosgw. 2021-02-11T06:09:57.080 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../111-radosgw_17.0.0-711-g84a8025e-1bionic_amd64.deb ... 2021-02-11T06:09:57.080 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking radosgw (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:09:57.080 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package rbd-fuse. 2021-02-11T06:09:57.080 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../112-rbd-fuse_17.0.0-711-g84a8025e-1bionic_amd64.deb ... 2021-02-11T06:09:57.080 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking rbd-fuse (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:09:57.081 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-logutils (0.3.3-5) ... 2021-02-11T06:09:57.081 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-werkzeug (0.14.1+dfsg1-1ubuntu0.1) ... 2021-02-11T06:09:57.081 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up librdkafka1:amd64 (0.11.3-1build1) ... 2021-02-11T06:09:57.081 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2021-02-11T06:09:57.081 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-pil:amd64 (5.1.0-1ubuntu0.4) ... 2021-02-11T06:09:57.081 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-simplegeneric (0.8.1-1) ... 2021-02-11T06:09:57.081 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-webencodings (0.5-2) ... 2021-02-11T06:09:57.081 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python-attr (17.4.0-2) ... 2021-02-11T06:09:57.081 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-cffi-backend (1.11.5-1) ... 2021-02-11T06:09:57.081 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-py (1.5.2-1) ... 2021-02-11T06:09:57.081 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-waitress (1.0.1-1) ... 2021-02-11T06:09:57.081 INFO:teuthology.orchestra.run.smithi061.stdout:update-alternatives: using /usr/bin/waitress-serve-python3 to provide /usr/bin/waitress-serve (waitress-serve) in auto mode 2021-02-11T06:09:57.081 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-olefile (0.45.1-1) ... 2021-02-11T06:09:57.081 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-ceph-argparse (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:09:57.081 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-tempita (0.5.2-2) ... 2021-02-11T06:09:57.082 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up socat (1.7.3.2-2ubuntu2) ... 2021-02-11T06:09:57.082 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-numpy (1:1.13.3-2ubuntu1) ... 2021-02-11T06:09:57.082 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../111-radosgw_17.0.0-711-g84a8025e-1bionic_amd64.deb ... 2021-02-11T06:09:57.082 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking radosgw (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:09:57.082 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package rbd-fuse. 2021-02-11T06:09:57.082 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../112-rbd-fuse_17.0.0-711-g84a8025e-1bionic_amd64.deb ... 2021-02-11T06:09:57.082 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking rbd-fuse (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:09:57.082 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-logutils (0.3.3-5) ... 2021-02-11T06:09:57.082 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-werkzeug (0.14.1+dfsg1-1ubuntu0.1) ... 2021-02-11T06:09:57.082 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up librdkafka1:amd64 (0.11.3-1build1) ... 2021-02-11T06:09:57.082 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2021-02-11T06:09:57.082 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-pil:amd64 (5.1.0-1ubuntu0.4) ... 2021-02-11T06:09:57.082 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-simplegeneric (0.8.1-1) ... 2021-02-11T06:09:57.083 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-webencodings (0.5-2) ... 2021-02-11T06:09:57.083 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python-attr (17.4.0-2) ... 2021-02-11T06:09:57.083 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-cffi-backend (1.11.5-1) ... 2021-02-11T06:09:57.083 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-py (1.5.2-1) ... 2021-02-11T06:09:57.083 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-waitress (1.0.1-1) ... 2021-02-11T06:09:57.083 INFO:teuthology.orchestra.run.smithi172.stdout:update-alternatives: using /usr/bin/waitress-serve-python3 to provide /usr/bin/waitress-serve (waitress-serve) in auto mode 2021-02-11T06:09:57.083 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-olefile (0.45.1-1) ... 2021-02-11T06:09:57.083 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-ceph-argparse (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:09:57.083 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-tempita (0.5.2-2) ... 2021-02-11T06:09:57.083 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up socat (1.7.3.2-2ubuntu2) ... 2021-02-11T06:09:57.083 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-numpy (1:1.13.3-2ubuntu1) ... 2021-02-11T06:09:57.083 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-webob (1:1.7.3-2fakesync1) ... 2021-02-11T06:09:57.083 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-bcrypt (3.1.4-2) ... 2021-02-11T06:09:57.083 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python-py (1.5.2-1) ... 2021-02-11T06:09:57.227 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-simplejson (3.13.2-1) ... 2021-02-11T06:09:57.314 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-webob (1:1.7.3-2fakesync1) ... 2021-02-11T06:09:57.562 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-json-pointer (1.10-1) ... 2021-02-11T06:09:57.641 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-bcrypt (3.1.4-2) ... 2021-02-11T06:09:57.651 INFO:teuthology.orchestra.run.smithi172.stdout:update-alternatives: using /usr/bin/python3-jsonpointer to provide /usr/bin/jsonpointer (jsonpointer) in auto mode 2021-02-11T06:09:57.905 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libonig4:amd64 (6.7.0-1) ... 2021-02-11T06:09:57.950 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python-py (1.5.2-1) ... 2021-02-11T06:09:58.031 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python-pluggy (0.6.0-1) ... 2021-02-11T06:09:58.177 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-simplejson (3.13.2-1) ... 2021-02-11T06:09:58.232 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-singledispatch (3.4.0.3-2) ... 2021-02-11T06:09:58.511 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-json-pointer (1.10-1) ... 2021-02-11T06:09:58.534 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-pyparsing (2.2.0+dfsg1-2) ... 2021-02-11T06:09:58.601 INFO:teuthology.orchestra.run.smithi061.stdout:update-alternatives: using /usr/bin/python3-jsonpointer to provide /usr/bin/jsonpointer (jsonpointer) in auto mode 2021-02-11T06:09:58.838 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up libonig4:amd64 (6.7.0-1) ... 2021-02-11T06:09:58.851 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-cycler (0.10.0-1) ... 2021-02-11T06:09:58.972 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python-pluggy (0.6.0-1) ... 2021-02-11T06:09:59.153 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-cherrypy3 (8.9.1-2) ... 2021-02-11T06:09:59.175 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-singledispatch (3.4.0.3-2) ... 2021-02-11T06:09:59.501 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-pyparsing (2.2.0+dfsg1-2) ... 2021-02-11T06:10:15.695 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-bs4 (4.6.0-1) ... 2021-02-11T06:10:15.695 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-pluggy (0.6.0-1) ... 2021-02-11T06:10:15.695 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up cephadm (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:10:15.695 INFO:teuthology.orchestra.run.smithi172.stdout:Adding system user cephadm....done 2021-02-11T06:10:15.695 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-markupsafe (1.0-1build1) ... 2021-02-11T06:10:15.696 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2021-02-11T06:10:15.696 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up nvme-cli (1.5-1ubuntu1.1) ... 2021-02-11T06:10:15.696 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-asn1crypto (0.24.0-1) ... 2021-02-11T06:10:15.696 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-prettytable (0.7.2-3) ... 2021-02-11T06:10:15.696 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libjq1:amd64 (1.5+dfsg-2) ... 2021-02-11T06:10:15.696 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-pyasn1 (0.4.2-3) ... 2021-02-11T06:10:15.696 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-ceph-common (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:10:15.696 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-monotonic (1.1-2) ... 2021-02-11T06:10:15.696 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libsnappy1v5:amd64 (1.1.7-1) ... 2021-02-11T06:10:15.696 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2021-02-11T06:10:15.696 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-html5lib (0.999999999-1) ... 2021-02-11T06:10:15.696 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2021-02-11T06:10:15.696 INFO:teuthology.orchestra.run.smithi172.stdout:update-alternatives: using /usr/bin/python3-jsondiff to provide /usr/bin/jsondiff (jsondiff) in auto mode 2021-02-11T06:10:15.697 INFO:teuthology.orchestra.run.smithi172.stdout:update-alternatives: using /usr/bin/python3-jsonpatch to provide /usr/bin/jsonpatch (jsonpatch) in auto mode 2021-02-11T06:10:15.697 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-repoze.lru (0.7-1) ... 2021-02-11T06:10:15.697 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2021-02-11T06:10:15.697 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2) ... 2021-02-11T06:10:15.697 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-attr (17.4.0-2) ... 2021-02-11T06:10:15.697 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python-funcsigs (1.0.2-4) ... 2021-02-11T06:10:15.697 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libjaeger (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:10:15.697 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-paste (2.0.3+dfsg-4ubuntu1) ... 2021-02-11T06:10:15.697 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-websocket (0.44.0-0ubuntu2) ... 2021-02-11T06:10:15.697 INFO:teuthology.orchestra.run.smithi172.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2021-02-11T06:10:15.697 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up liboath0 (2.6.1-1) ... 2021-02-11T06:10:15.697 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2021-02-11T06:10:15.697 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-tk:amd64 (3.6.9-1~18.04) ... 2021-02-11T06:10:15.697 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python-pastedeploy-tpl (1.5.2-4) ... 2021-02-11T06:10:15.698 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-setuptools (39.0.1-2) ... 2021-02-11T06:10:15.698 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up liblttng-ust0:amd64 (2.10.1-1) ... 2021-02-11T06:10:15.698 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-decorator (4.1.2-1) ... 2021-02-11T06:10:15.698 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-tz (2018.3-2) ... 2021-02-11T06:10:15.698 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-uritemplate (0.6-1ubuntu1) ... 2021-02-11T06:10:15.698 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-jwt (1.5.3+ds1-1) ... 2021-02-11T06:10:15.698 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-joblib (0.11-1) ... 2021-02-11T06:10:15.698 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-routes (2.4.1-1) ... 2021-02-11T06:10:15.698 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-pyinotify (0.9.6-1) ... 2021-02-11T06:10:15.698 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-dateutil (2.6.1-1) ... 2021-02-11T06:10:15.698 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-cryptography (2.1.4-1ubuntu1.4) ... 2021-02-11T06:10:15.698 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-fasteners (0.12.0-3) ... 2021-02-11T06:10:15.698 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up jq (1.5+dfsg-2) ... 2021-02-11T06:10:15.699 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-scipy (0.19.1-2ubuntu1) ... 2021-02-11T06:10:15.699 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-lxml:amd64 (4.2.1-1ubuntu0.3) ... 2021-02-11T06:10:15.699 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-mako (1.0.7+ds1-1) ... 2021-02-11T06:10:15.699 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-sklearn-lib (0.19.1-3) ... 2021-02-11T06:10:15.699 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-pytest (3.3.2-2) ... 2021-02-11T06:10:15.699 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python-pytest (3.3.2-2) ... 2021-02-11T06:10:15.699 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libleveldb1v5:amd64 (1.20-2) ... 2021-02-11T06:10:15.699 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up xmlstarlet (1.6.1-2) ... 2021-02-11T06:10:15.699 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up librados2 (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:10:15.699 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-rsa (3.4.2-1) ... 2021-02-11T06:10:15.699 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libcephfs2 (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:10:15.699 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libcephfs-dev (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:10:15.699 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-rados (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:10:15.699 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-openssl (17.5.0-1ubuntu1) ... 2021-02-11T06:10:15.700 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-cycler (0.10.0-1) ... 2021-02-11T06:10:15.700 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-cherrypy3 (8.9.1-2) ... 2021-02-11T06:10:15.700 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-bs4 (4.6.0-1) ... 2021-02-11T06:10:15.700 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-pluggy (0.6.0-1) ... 2021-02-11T06:10:15.700 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up cephadm (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:10:15.700 INFO:teuthology.orchestra.run.smithi061.stdout:Adding system user cephadm....done 2021-02-11T06:10:15.700 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-markupsafe (1.0-1build1) ... 2021-02-11T06:10:15.700 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2021-02-11T06:10:15.700 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up nvme-cli (1.5-1ubuntu1.1) ... 2021-02-11T06:10:15.700 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-asn1crypto (0.24.0-1) ... 2021-02-11T06:10:15.700 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-prettytable (0.7.2-3) ... 2021-02-11T06:10:15.700 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up libjq1:amd64 (1.5+dfsg-2) ... 2021-02-11T06:10:15.700 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-pyasn1 (0.4.2-3) ... 2021-02-11T06:10:15.700 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-ceph-common (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:10:15.701 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-monotonic (1.1-2) ... 2021-02-11T06:10:15.701 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up libsnappy1v5:amd64 (1.1.7-1) ... 2021-02-11T06:10:15.701 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2021-02-11T06:10:15.701 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-html5lib (0.999999999-1) ... 2021-02-11T06:10:15.701 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2021-02-11T06:10:15.701 INFO:teuthology.orchestra.run.smithi061.stdout:update-alternatives: using /usr/bin/python3-jsondiff to provide /usr/bin/jsondiff (jsondiff) in auto mode 2021-02-11T06:10:15.701 INFO:teuthology.orchestra.run.smithi061.stdout:update-alternatives: using /usr/bin/python3-jsonpatch to provide /usr/bin/jsonpatch (jsonpatch) in auto mode 2021-02-11T06:10:15.701 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-repoze.lru (0.7-1) ... 2021-02-11T06:10:15.701 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2021-02-11T06:10:15.701 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2) ... 2021-02-11T06:10:15.701 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-attr (17.4.0-2) ... 2021-02-11T06:10:15.701 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python-funcsigs (1.0.2-4) ... 2021-02-11T06:10:15.701 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up libjaeger (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:10:15.702 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-paste (2.0.3+dfsg-4ubuntu1) ... 2021-02-11T06:10:15.702 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-websocket (0.44.0-0ubuntu2) ... 2021-02-11T06:10:15.702 INFO:teuthology.orchestra.run.smithi061.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2021-02-11T06:10:15.702 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up liboath0 (2.6.1-1) ... 2021-02-11T06:10:15.702 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2021-02-11T06:10:15.702 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-tk:amd64 (3.6.9-1~18.04) ... 2021-02-11T06:10:15.702 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python-pastedeploy-tpl (1.5.2-4) ... 2021-02-11T06:10:15.702 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-setuptools (39.0.1-2) ... 2021-02-11T06:10:15.702 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up liblttng-ust0:amd64 (2.10.1-1) ... 2021-02-11T06:10:15.702 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-decorator (4.1.2-1) ... 2021-02-11T06:10:15.702 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-tz (2018.3-2) ... 2021-02-11T06:10:15.702 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-uritemplate (0.6-1ubuntu1) ... 2021-02-11T06:10:15.702 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-jwt (1.5.3+ds1-1) ... 2021-02-11T06:10:15.703 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-joblib (0.11-1) ... 2021-02-11T06:10:15.703 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-routes (2.4.1-1) ... 2021-02-11T06:10:15.703 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-pyinotify (0.9.6-1) ... 2021-02-11T06:10:15.703 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-dateutil (2.6.1-1) ... 2021-02-11T06:10:15.703 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-cryptography (2.1.4-1ubuntu1.4) ... 2021-02-11T06:10:15.703 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-fasteners (0.12.0-3) ... 2021-02-11T06:10:15.703 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up jq (1.5+dfsg-2) ... 2021-02-11T06:10:15.703 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-scipy (0.19.1-2ubuntu1) ... 2021-02-11T06:10:15.703 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-lxml:amd64 (4.2.1-1ubuntu0.3) ... 2021-02-11T06:10:15.703 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-mako (1.0.7+ds1-1) ... 2021-02-11T06:10:15.703 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-sklearn-lib (0.19.1-3) ... 2021-02-11T06:10:15.703 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-pytest (3.3.2-2) ... 2021-02-11T06:10:15.703 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python-pytest (3.3.2-2) ... 2021-02-11T06:10:15.703 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up libleveldb1v5:amd64 (1.20-2) ... 2021-02-11T06:10:15.704 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up xmlstarlet (1.6.1-2) ... 2021-02-11T06:10:15.704 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up librados2 (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:10:15.704 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-rsa (3.4.2-1) ... 2021-02-11T06:10:15.704 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up libcephfs2 (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:10:15.704 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up libcephfs-dev (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:10:15.746 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-pastedeploy (1.5.2-4) ... 2021-02-11T06:10:15.769 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-rados (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:10:15.869 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-openssl (17.5.0-1ubuntu1) ... 2021-02-11T06:10:16.072 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-oauth2client (4.1.2-3) ... 2021-02-11T06:10:16.162 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-pastedeploy (1.5.2-4) ... 2021-02-11T06:10:16.447 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-oauth2client (4.1.2-3) ... 2021-02-11T06:10:16.509 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-matplotlib (2.1.1-2ubuntu3) ... 2021-02-11T06:10:16.852 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-matplotlib (2.1.1-2ubuntu3) ... 2021-02-11T06:10:17.364 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-sklearn (0.19.1-3) ... 2021-02-11T06:10:17.696 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-sklearn (0.19.1-3) ... 2021-02-11T06:10:18.172 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-webtest (2.0.28-1ubuntu1) ... 2021-02-11T06:10:18.445 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libradosstriper1 (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:10:18.535 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-webtest (2.0.28-1ubuntu1) ... 2021-02-11T06:10:18.571 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-cephfs (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:10:18.697 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up librgw2 (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:10:18.823 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-fuse (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:10:18.836 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up libradosstriper1 (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:10:18.937 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-cephfs (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:10:19.035 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2021-02-11T06:10:19.036 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2021-02-11T06:10:19.037 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up librgw2 (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:10:19.166 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up ceph-fuse (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:10:19.383 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up librbd1 (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:10:19.397 INFO:teuthology.orchestra.run.smithi061.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2021-02-11T06:10:19.397 INFO:teuthology.orchestra.run.smithi061.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2021-02-11T06:10:19.492 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-pecan (1.2.1-2) ... 2021-02-11T06:10:19.566 INFO:teuthology.orchestra.run.smithi172.stdout:update-alternatives: using /usr/bin/python3-pecan to provide /usr/bin/pecan (pecan) in auto mode 2021-02-11T06:10:19.598 INFO:teuthology.orchestra.run.smithi172.stdout:update-alternatives: using /usr/bin/python3-gunicorn_pecan to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode 2021-02-11T06:10:19.660 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up librbd1 (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:10:19.777 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-pecan (1.2.1-2) ... 2021-02-11T06:10:19.850 INFO:teuthology.orchestra.run.smithi061.stdout:update-alternatives: using /usr/bin/python3-pecan to provide /usr/bin/pecan (pecan) in auto mode 2021-02-11T06:10:19.882 INFO:teuthology.orchestra.run.smithi061.stdout:update-alternatives: using /usr/bin/python3-gunicorn_pecan to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode 2021-02-11T06:10:19.887 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-kubernetes (2.0.0-2ubuntu1) ... 2021-02-11T06:10:20.172 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-kubernetes (2.0.0-2ubuntu1) ... 2021-02-11T06:10:20.590 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-mgr-modules-core (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:10:20.699 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-pastescript (2.0.2-2) ... 2021-02-11T06:10:20.850 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up ceph-mgr-modules-core (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:10:25.760 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-rgw (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:10:25.760 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up rbd-fuse (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:10:25.760 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-rbd (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:10:25.760 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-common (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:10:25.760 INFO:teuthology.orchestra.run.smithi172.stdout:Adding group ceph....done 2021-02-11T06:10:25.760 INFO:teuthology.orchestra.run.smithi172.stdout:Adding system user ceph....done 2021-02-11T06:10:25.761 INFO:teuthology.orchestra.run.smithi172.stdout:Setting system user ceph properties....done 2021-02-11T06:10:25.761 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2021-02-11T06:10:25.761 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2021-02-11T06:10:25.761 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up radosgw (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:10:25.761 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2021-02-11T06:10:25.761 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2021-02-11T06:10:25.761 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-test (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:10:25.761 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-base (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:10:25.761 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2021-02-11T06:10:25.761 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-mgr (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:10:25.761 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2021-02-11T06:10:25.761 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2021-02-11T06:10:25.761 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-mgr-diskprediction-local (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:10:25.761 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-osd (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:10:25.762 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2021-02-11T06:10:25.762 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2021-02-11T06:10:25.762 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-pastescript (2.0.2-2) ... 2021-02-11T06:10:25.762 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-rgw (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:10:25.762 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up rbd-fuse (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:10:25.762 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-rbd (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:10:25.762 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up ceph-common (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:10:25.762 INFO:teuthology.orchestra.run.smithi061.stdout:Adding group ceph....done 2021-02-11T06:10:25.762 INFO:teuthology.orchestra.run.smithi061.stdout:Adding system user ceph....done 2021-02-11T06:10:25.762 INFO:teuthology.orchestra.run.smithi061.stdout:Setting system user ceph properties....done 2021-02-11T06:10:25.762 INFO:teuthology.orchestra.run.smithi061.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2021-02-11T06:10:25.762 INFO:teuthology.orchestra.run.smithi061.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2021-02-11T06:10:25.762 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up radosgw (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:10:25.763 INFO:teuthology.orchestra.run.smithi061.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2021-02-11T06:10:25.763 INFO:teuthology.orchestra.run.smithi061.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2021-02-11T06:10:25.763 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up ceph-test (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:10:25.763 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up ceph-base (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:10:25.763 INFO:teuthology.orchestra.run.smithi061.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2021-02-11T06:10:25.763 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up ceph-mgr (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:10:25.763 INFO:teuthology.orchestra.run.smithi061.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2021-02-11T06:10:25.763 INFO:teuthology.orchestra.run.smithi061.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2021-02-11T06:10:25.763 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up ceph-mgr-diskprediction-local (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:10:25.763 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up ceph-osd (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:10:25.766 INFO:teuthology.orchestra.run.smithi061.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2021-02-11T06:10:25.766 INFO:teuthology.orchestra.run.smithi061.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2021-02-11T06:10:25.824 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-mgr-cephadm (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:10:25.958 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-mgr-rook (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:10:26.059 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up ceph-mgr-cephadm (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:10:26.194 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-mds (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:10:26.201 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up ceph-mgr-rook (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:10:26.344 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up ceph-mds (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:10:26.388 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2021-02-11T06:10:26.388 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2021-02-11T06:10:26.556 INFO:teuthology.orchestra.run.smithi061.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2021-02-11T06:10:26.556 INFO:teuthology.orchestra.run.smithi061.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2021-02-11T06:10:26.653 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-mon (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:10:26.847 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2021-02-11T06:10:26.847 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2021-02-11T06:10:27.029 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up ceph-mon (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:10:27.130 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-mgr-k8sevents (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:10:27.234 INFO:teuthology.orchestra.run.smithi061.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2021-02-11T06:10:27.235 INFO:teuthology.orchestra.run.smithi061.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2021-02-11T06:10:27.255 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-mgr-dashboard (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:10:27.398 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:10:27.508 INFO:teuthology.orchestra.run.smithi172.stdout:Processing triggers for systemd (237-3ubuntu10.44) ... 2021-02-11T06:10:27.522 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up ceph-mgr-k8sevents (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:10:27.631 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up ceph-mgr-dashboard (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:10:27.701 INFO:teuthology.orchestra.run.smithi172.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2021-02-11T06:10:27.749 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up ceph (17.0.0-711-g84a8025e-1bionic) ... 2021-02-11T06:10:27.876 INFO:teuthology.orchestra.run.smithi061.stdout:Processing triggers for systemd (237-3ubuntu10.44) ... 2021-02-11T06:10:28.069 INFO:teuthology.orchestra.run.smithi061.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2021-02-11T06:10:29.763 INFO:teuthology.orchestra.run.smithi172.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2021-02-11T06:10:29.805 INFO:teuthology.orchestra.run.smithi172.stdout:ureadahead will be reprofiled on next reboot 2021-02-11T06:10:29.855 INFO:teuthology.orchestra.run.smithi172.stdout:Processing triggers for libc-bin (2.27-3ubuntu1.4) ... 2021-02-11T06:10:30.196 INFO:teuthology.orchestra.run.smithi061.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2021-02-11T06:10:30.230 INFO:teuthology.orchestra.run.smithi061.stdout:ureadahead will be reprofiled on next reboot 2021-02-11T06:10:30.280 INFO:teuthology.orchestra.run.smithi061.stdout:Processing triggers for libc-bin (2.27-3ubuntu1.4) ... 2021-02-11T06:10:30.334 INFO:teuthology.orchestra.run.smithi172.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2021-02-11T06:10:30.337 DEBUG:teuthology.parallel:result is None 2021-02-11T06:10:30.723 INFO:teuthology.orchestra.run.smithi061.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2021-02-11T06:10:30.725 DEBUG:teuthology.parallel:result is None 2021-02-11T06:10:30.726 DEBUG:teuthology.orchestra.run.smithi061:> sudo lsb_release -is 2021-02-11T06:10:30.774 INFO:teuthology.orchestra.run.smithi061.stdout:Ubuntu 2021-02-11T06:10:30.779 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-02-11T06:10:30.780 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=84a8025e5a892a6879f75e17e03444bf443eae4b 2021-02-11T06:10:30.965 DEBUG:teuthology.orchestra.run.smithi061:> dpkg-query -W -f '${Version}' ceph 2021-02-11T06:10:30.989 INFO:teuthology.orchestra.run.smithi061.stdout:17.0.0-711-g84a8025e-1bionic 2021-02-11T06:10:30.990 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-711-g84a8025e-1bionic 2021-02-11T06:10:30.990 INFO:teuthology.task.install:The correct ceph version 17.0.0-711-g84a8025e-1bionic is installed. 2021-02-11T06:10:30.990 DEBUG:teuthology.orchestra.run.smithi172:> sudo lsb_release -is 2021-02-11T06:10:31.045 INFO:teuthology.orchestra.run.smithi172.stdout:Ubuntu 2021-02-11T06:10:31.051 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-02-11T06:10:31.051 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=84a8025e5a892a6879f75e17e03444bf443eae4b 2021-02-11T06:10:31.215 DEBUG:teuthology.orchestra.run.smithi172:> dpkg-query -W -f '${Version}' ceph 2021-02-11T06:10:31.241 INFO:teuthology.orchestra.run.smithi172.stdout:17.0.0-711-g84a8025e-1bionic 2021-02-11T06:10:31.241 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-711-g84a8025e-1bionic 2021-02-11T06:10:31.242 INFO:teuthology.task.install:The correct ceph version 17.0.0-711-g84a8025e-1bionic is installed. 2021-02-11T06:10:31.242 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2021-02-11T06:10:31.248 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2021-02-11T06:10:31.248 DEBUG:teuthology.orchestra.run.smithi061:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-02-11T06:10:31.263 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2021-02-11T06:10:31.264 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-02-11T06:10:31.296 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2021-02-11T06:10:37.869 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2021-02-11T06:10:37.869 DEBUG:teuthology.orchestra.run.smithi061:> sudo dd of=/usr/bin/daemon-helper 2021-02-11T06:10:37.885 DEBUG:teuthology.orchestra.run.smithi061:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-02-11T06:10:37.942 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2021-02-11T06:10:37.942 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/usr/bin/daemon-helper 2021-02-11T06:10:37.957 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-02-11T06:10:38.013 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2021-02-11T06:10:38.013 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2021-02-11T06:10:38.013 DEBUG:teuthology.orchestra.run.smithi061:> sudo dd of=/usr/bin/adjust-ulimits 2021-02-11T06:10:38.028 DEBUG:teuthology.orchestra.run.smithi061:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-02-11T06:10:38.085 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2021-02-11T06:10:38.085 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/usr/bin/adjust-ulimits 2021-02-11T06:10:38.100 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-02-11T06:10:38.157 INFO:teuthology.run_tasks:Running task ceph... 2021-02-11T06:10:38.242 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2021-02-11T06:10:38.243 DEBUG:teuthology.orchestra.run.smithi061:> sudo chmod 777 /var/log/ceph 2021-02-11T06:10:38.245 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 777 /var/log/ceph 2021-02-11T06:10:38.259 INFO:tasks.ceph:Disabling ceph logrotate... 2021-02-11T06:10:38.259 DEBUG:teuthology.orchestra.run.smithi061:> sudo rm -f -- /etc/logrotate.d/ceph 2021-02-11T06:10:38.301 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f -- /etc/logrotate.d/ceph 2021-02-11T06:10:38.317 INFO:tasks.ceph:Creating extra log directories... 2021-02-11T06:10:38.317 DEBUG:teuthology.orchestra.run.smithi061:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2021-02-11T06:10:38.357 DEBUG:teuthology.orchestra.run.smithi172:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2021-02-11T06:10:38.661 INFO:tasks.ceph:Creating ceph cluster c1... 2021-02-11T06:10:38.662 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'cpu_profile': set(), 'cluster': 'c1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2021-02-11T06:10:38.662 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/aemerson-2021-02-11_05:45:06-rgw-wip-easter-bunny-distro-basic-smithi/5874443', 'branch': 'wip-easter-bunny', 'description': 'rgw/multisite/{clusters frontend/beast omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind}', 'email': 'aemerson@redhat.com', 'first_in_suite': False, 'job_id': '5874443', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'aemerson-2021-02-11_05:45:06-rgw-wip-easter-bunny-distro-basic-smithi', 'nuke-on-error': True, 'os_type': 'ubuntu', 'overrides': {'admin_socket': {'branch': 'wip-easter-bunny'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '84a8025e5a892a6879f75e17e03444bf443eae4b', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': '84a8025e5a892a6879f75e17e03444bf443eae4b'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}, {'endpoints': ['c2.client.1'], 'is_pubsub': True, 'name': 'test-zone4'}]}]}, 'workunit': {'branch': 'wip-easter-bunny', 'sha1': '84a8025e5a892a6879f75e17e03444bf443eae4b'}}, 'owner': 'scheduled_aemerson@teuthology', 'priority': 70, 'repo': 'git://git.ceph.com/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'sha1': '84a8025e5a892a6879f75e17e03444bf443eae4b', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'wip-easter-bunny', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_84a8025e5a892a6879f75e17e03444bf443eae4b/qa', 'suite_relpath': 'qa', 'suite_repo': 'git://git.ceph.com/ceph-ci.git', 'suite_sha1': '84a8025e5a892a6879f75e17e03444bf443eae4b', 'targets': {'smithi061.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9nFZMnqNQMV9nVvBUk/hdcUhFPn2XCz9ZkCiQbtge7HeikSAfUCyOCHSfz0S1pmOcIbc6LkEm4xWkAeLB9/pdSZ+PzVLBUanYR6RkuRKO4xTYn+JavSZmJCi3NQ5NcsBR04RRvp/saikmxQmmdlwsgajmkmavyzkL90CEquTby7jL4PERDRhH/qpnwlzPWgaaiea3k9Yh41Jvtd1xQi0+1IsNGsmoVgvfz90gsgnXHmuuYXUTC9xbXtlZnq1pOmBAtWlaf0n/YpZUMki+jsM6TOwG3OOGZ2m69P7yNNbjhZiBmDSi0lBcx7cMG7lyYWJrFO+6a+MvQA1L/Z1hR14X', 'smithi172.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKc1kIvUvofa8RKQkSgMIYhvCE2w7RZG4ZgcVx5kPL/pTrHdUfID6cH6NblNsgR97jYHYb7olU29Qx8ZIgSSI3zbdAxGNt1IgvEfG58vL1U2g0Ln3iIV/ZRIzyl22IbRjb4UNstYPCxc5lX9Ni+ngwwE7XmvflhXJAL5Ykpq74qpcvhcG0T/JCADGX2q+LhAUSmgb6vYzWdwz1GhObmV2W80cMQBpmsoffW5w3w7uu6rHdNA0rK1C9x0/bqDMp1TUViv3pe7jxD3JlufcDDIszdtwc2s4n1w0HhzbQmV1tJZUD1tKjtTOoqWuFdMb9t//7ZKqjatFUjJzKc3ZWv86X'}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '84a8025e5a892a6879f75e17e03444bf443eae4b', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2'}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology_branch': 'master', 'timestamp': '2021-02-11_05:45:06', 'tube': 'smithi', 'user': 'aemerson', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.29888'} 2021-02-11T06:10:38.662 DEBUG:teuthology.orchestra.run.smithi061:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2021-02-11T06:10:38.665 DEBUG:teuthology.orchestra.run.smithi172:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2021-02-11T06:10:38.671 DEBUG:teuthology.orchestra.run.smithi061:> sudo install -d -m0777 -- /var/run/ceph 2021-02-11T06:10:38.713 DEBUG:teuthology.orchestra.run.smithi172:> sudo install -d -m0777 -- /var/run/ceph 2021-02-11T06:10:38.729 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2021-02-11T06:10:38.729 DEBUG:teuthology.orchestra.run.smithi061:> dd if=/scratch_devs of=/dev/stdout 2021-02-11T06:10:38.773 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-02-11T06:10:38.773 DEBUG:teuthology.orchestra.run.smithi061:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-02-11T06:10:38.821 INFO:teuthology.orchestra.run.smithi061.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2021-02-11T06:10:38.822 INFO:teuthology.orchestra.run.smithi061.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-11T06:10:38.822 INFO:teuthology.orchestra.run.smithi061.stdout:Device: 6h/6d Inode: 340 Links: 1 2021-02-11T06:10:38.822 INFO:teuthology.orchestra.run.smithi061.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-11T06:10:38.822 INFO:teuthology.orchestra.run.smithi061.stdout:Access: 2021-02-11 06:07:02.202612848 +0000 2021-02-11T06:10:38.822 INFO:teuthology.orchestra.run.smithi061.stdout:Modify: 2021-02-11 06:07:02.074612849 +0000 2021-02-11T06:10:38.822 INFO:teuthology.orchestra.run.smithi061.stdout:Change: 2021-02-11 06:07:02.074612849 +0000 2021-02-11T06:10:38.822 INFO:teuthology.orchestra.run.smithi061.stdout: Birth: - 2021-02-11T06:10:38.831 INFO:teuthology.orchestra.run.smithi061.stderr:1+0 records in 2021-02-11T06:10:38.831 INFO:teuthology.orchestra.run.smithi061.stderr:1+0 records out 2021-02-11T06:10:38.831 INFO:teuthology.orchestra.run.smithi061.stderr:512 bytes copied, 0.000336763 s, 1.5 MB/s 2021-02-11T06:10:38.835 DEBUG:teuthology.orchestra.run.smithi061:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-02-11T06:10:38.886 INFO:teuthology.orchestra.run.smithi061.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2021-02-11T06:10:38.886 INFO:teuthology.orchestra.run.smithi061.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-11T06:10:38.886 INFO:teuthology.orchestra.run.smithi061.stdout:Device: 6h/6d Inode: 346 Links: 1 2021-02-11T06:10:38.886 INFO:teuthology.orchestra.run.smithi061.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-11T06:10:38.886 INFO:teuthology.orchestra.run.smithi061.stdout:Access: 2021-02-11 06:07:02.490612846 +0000 2021-02-11T06:10:38.886 INFO:teuthology.orchestra.run.smithi061.stdout:Modify: 2021-02-11 06:07:02.350612847 +0000 2021-02-11T06:10:38.886 INFO:teuthology.orchestra.run.smithi061.stdout:Change: 2021-02-11 06:07:02.350612847 +0000 2021-02-11T06:10:38.886 INFO:teuthology.orchestra.run.smithi061.stdout: Birth: - 2021-02-11T06:10:38.895 INFO:teuthology.orchestra.run.smithi061.stderr:1+0 records in 2021-02-11T06:10:38.895 INFO:teuthology.orchestra.run.smithi061.stderr:1+0 records out 2021-02-11T06:10:38.896 INFO:teuthology.orchestra.run.smithi061.stderr:512 bytes copied, 0.00039249 s, 1.3 MB/s 2021-02-11T06:10:38.900 DEBUG:teuthology.orchestra.run.smithi061:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-02-11T06:10:38.949 INFO:teuthology.orchestra.run.smithi061.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2021-02-11T06:10:38.950 INFO:teuthology.orchestra.run.smithi061.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-11T06:10:38.950 INFO:teuthology.orchestra.run.smithi061.stdout:Device: 6h/6d Inode: 353 Links: 1 2021-02-11T06:10:38.950 INFO:teuthology.orchestra.run.smithi061.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-11T06:10:38.950 INFO:teuthology.orchestra.run.smithi061.stdout:Access: 2021-02-11 06:07:02.714612845 +0000 2021-02-11T06:10:38.950 INFO:teuthology.orchestra.run.smithi061.stdout:Modify: 2021-02-11 06:07:02.610612846 +0000 2021-02-11T06:10:38.950 INFO:teuthology.orchestra.run.smithi061.stdout:Change: 2021-02-11 06:07:02.610612846 +0000 2021-02-11T06:10:38.950 INFO:teuthology.orchestra.run.smithi061.stdout: Birth: - 2021-02-11T06:10:38.959 INFO:teuthology.orchestra.run.smithi061.stderr:1+0 records in 2021-02-11T06:10:38.959 INFO:teuthology.orchestra.run.smithi061.stderr:1+0 records out 2021-02-11T06:10:38.959 INFO:teuthology.orchestra.run.smithi061.stderr:512 bytes copied, 0.000414187 s, 1.2 MB/s 2021-02-11T06:10:38.964 DEBUG:teuthology.orchestra.run.smithi061:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-02-11T06:10:39.017 INFO:teuthology.orchestra.run.smithi061.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2021-02-11T06:10:39.017 INFO:teuthology.orchestra.run.smithi061.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-11T06:10:39.018 INFO:teuthology.orchestra.run.smithi061.stdout:Device: 6h/6d Inode: 359 Links: 1 2021-02-11T06:10:39.018 INFO:teuthology.orchestra.run.smithi061.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-11T06:10:39.018 INFO:teuthology.orchestra.run.smithi061.stdout:Access: 2021-02-11 06:07:02.974612844 +0000 2021-02-11T06:10:39.018 INFO:teuthology.orchestra.run.smithi061.stdout:Modify: 2021-02-11 06:07:02.866612844 +0000 2021-02-11T06:10:39.018 INFO:teuthology.orchestra.run.smithi061.stdout:Change: 2021-02-11 06:07:02.866612844 +0000 2021-02-11T06:10:39.018 INFO:teuthology.orchestra.run.smithi061.stdout: Birth: - 2021-02-11T06:10:39.027 INFO:teuthology.orchestra.run.smithi061.stderr:1+0 records in 2021-02-11T06:10:39.027 INFO:teuthology.orchestra.run.smithi061.stderr:1+0 records out 2021-02-11T06:10:39.027 INFO:teuthology.orchestra.run.smithi061.stderr:512 bytes copied, 0.00041316 s, 1.2 MB/s 2021-02-11T06:10:39.032 INFO:tasks.ceph:osd dev map: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2021-02-11T06:10:39.032 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi061.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2021-02-11T06:10:39.032 INFO:tasks.ceph:Generating config... 2021-02-11T06:10:39.034 INFO:tasks.ceph:[client] debug rgw = 20 2021-02-11T06:10:39.035 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2021-02-11T06:10:39.035 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2021-02-11T06:10:39.035 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-02-11T06:10:39.035 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2021-02-11T06:10:39.035 INFO:tasks.ceph:[client] rgw data log num shards = 4 2021-02-11T06:10:39.035 INFO:tasks.ceph:[client] rgw md log max shards = 4 2021-02-11T06:10:39.035 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2021-02-11T06:10:39.035 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2021-02-11T06:10:39.035 INFO:tasks.ceph:[client] setgroup = ceph 2021-02-11T06:10:39.035 INFO:tasks.ceph:[client] setuser = ceph 2021-02-11T06:10:39.035 INFO:tasks.ceph:[global] osd heartbeat grace = 40 2021-02-11T06:10:39.035 INFO:tasks.ceph:[mgr] debug mgr = 20 2021-02-11T06:10:39.035 INFO:tasks.ceph:[mgr] debug ms = 1 2021-02-11T06:10:39.035 INFO:tasks.ceph:[mon] debug mon = 20 2021-02-11T06:10:39.036 INFO:tasks.ceph:[mon] debug ms = 1 2021-02-11T06:10:39.036 INFO:tasks.ceph:[mon] debug paxos = 20 2021-02-11T06:10:39.036 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2021-02-11T06:10:39.036 INFO:tasks.ceph:[osd] debug filestore = 20 2021-02-11T06:10:39.036 INFO:tasks.ceph:[osd] debug journal = 20 2021-02-11T06:10:39.036 INFO:tasks.ceph:[osd] debug ms = 20 2021-02-11T06:10:39.036 INFO:tasks.ceph:[osd] debug osd = 25 2021-02-11T06:10:39.036 INFO:tasks.ceph:[osd] osd fast shutdown = False 2021-02-11T06:10:39.036 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2021-02-11T06:10:39.036 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2021-02-11T06:10:39.036 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2021-02-11T06:10:39.036 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2021-02-11T06:10:39.037 INFO:tasks.ceph:Setting up c1.mon.a... 2021-02-11T06:10:39.037 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c1.keyring 2021-02-11T06:10:39.107 INFO:teuthology.orchestra.run.smithi061.stdout:creating /etc/ceph/c1.keyring 2021-02-11T06:10:39.109 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c1.keyring 2021-02-11T06:10:39.186 DEBUG:teuthology.orchestra.run.smithi061:> sudo chmod 0644 /etc/ceph/c1.keyring 2021-02-11T06:10:39.240 DEBUG:tasks.ceph:Ceph mon addresses: [('c1.mon.a', '172.21.15.61')] 2021-02-11T06:10:39.240 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.61', 'osd heartbeat grace': 40}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'mon osd crush smoke test': False}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2021-02-11T06:10:39.241 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2021-02-11T06:10:39.241 DEBUG:teuthology.orchestra.run.smithi061:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2021-02-11T06:10:39.289 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.61 --print /home/ubuntu/cephtest/c1.monmap 2021-02-11T06:10:39.361 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:10:39.362 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:10:39.362 INFO:teuthology.orchestra.run.smithi061.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c1.monmap 2021-02-11T06:10:39.362 INFO:teuthology.orchestra.run.smithi061.stdout:monmaptool: generated fsid 043de53e-a645-454b-bce7-c489302846ce 2021-02-11T06:10:39.362 INFO:teuthology.orchestra.run.smithi061.stdout:epoch 0 2021-02-11T06:10:39.362 INFO:teuthology.orchestra.run.smithi061.stdout:fsid 043de53e-a645-454b-bce7-c489302846ce 2021-02-11T06:10:39.362 INFO:teuthology.orchestra.run.smithi061.stdout:last_changed 2021-02-11T06:10:39.364465+0000 2021-02-11T06:10:39.362 INFO:teuthology.orchestra.run.smithi061.stdout:created 2021-02-11T06:10:39.364465+0000 2021-02-11T06:10:39.363 INFO:teuthology.orchestra.run.smithi061.stdout:min_mon_release 0 (unknown) 2021-02-11T06:10:39.363 INFO:teuthology.orchestra.run.smithi061.stdout:election_strategy: 1 2021-02-11T06:10:39.363 INFO:teuthology.orchestra.run.smithi061.stdout:0: [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] mon.a 2021-02-11T06:10:39.363 INFO:teuthology.orchestra.run.smithi061.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c1.monmap (1 monitors) 2021-02-11T06:10:39.364 DEBUG:teuthology.orchestra.run.smithi061:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2021-02-11T06:10:39.413 INFO:tasks.ceph:Writing /etc/ceph/c1.conf for FSID 043de53e-a645-454b-bce7-c489302846ce... 2021-02-11T06:10:39.413 DEBUG:teuthology.orchestra.run.smithi061:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2021-02-11T06:10:39.456 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2021-02-11T06:10:39.487 INFO:teuthology.orchestra.run.smithi172.stdout:[global] 2021-02-11T06:10:39.487 INFO:teuthology.orchestra.run.smithi172.stdout: chdir = "" 2021-02-11T06:10:39.487 INFO:teuthology.orchestra.run.smithi172.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-02-11T06:10:39.487 INFO:teuthology.orchestra.run.smithi172.stdout: auth supported = cephx 2021-02-11T06:10:39.487 INFO:teuthology.orchestra.run.smithi172.stdout: 2021-02-11T06:10:39.487 INFO:teuthology.orchestra.run.smithi172.stdout: filestore xattr use omap = true 2021-02-11T06:10:39.488 INFO:teuthology.orchestra.run.smithi172.stdout: 2021-02-11T06:10:39.488 INFO:teuthology.orchestra.run.smithi172.stdout: mon clock drift allowed = 1.000 2021-02-11T06:10:39.488 INFO:teuthology.orchestra.run.smithi172.stdout: 2021-02-11T06:10:39.488 INFO:teuthology.orchestra.run.smithi172.stdout: osd crush chooseleaf type = 0 2021-02-11T06:10:39.488 INFO:teuthology.orchestra.run.smithi172.stdout: auth debug = true 2021-02-11T06:10:39.488 INFO:teuthology.orchestra.run.smithi172.stdout: 2021-02-11T06:10:39.488 INFO:teuthology.orchestra.run.smithi172.stdout: ms die on old message = true 2021-02-11T06:10:39.488 INFO:teuthology.orchestra.run.smithi172.stdout: ms die on bug = true 2021-02-11T06:10:39.488 INFO:teuthology.orchestra.run.smithi172.stdout: 2021-02-11T06:10:39.488 INFO:teuthology.orchestra.run.smithi172.stdout: mon max pg per osd = 10000 # >= luminous 2021-02-11T06:10:39.488 INFO:teuthology.orchestra.run.smithi172.stdout: mon pg warn max object skew = 0 2021-02-11T06:10:39.488 INFO:teuthology.orchestra.run.smithi172.stdout: 2021-02-11T06:10:39.488 INFO:teuthology.orchestra.run.smithi172.stdout: # disable pg_autoscaler by default for new pools 2021-02-11T06:10:39.489 INFO:teuthology.orchestra.run.smithi172.stdout: osd_pool_default_pg_autoscale_mode = off 2021-02-11T06:10:39.489 INFO:teuthology.orchestra.run.smithi172.stdout: 2021-02-11T06:10:39.489 INFO:teuthology.orchestra.run.smithi172.stdout: osd pool default size = 2 2021-02-11T06:10:39.489 INFO:teuthology.orchestra.run.smithi172.stdout: 2021-02-11T06:10:39.489 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd allow primary affinity = true 2021-02-11T06:10:39.489 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd allow pg remap = true 2021-02-11T06:10:39.489 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on legacy crush tunables = false 2021-02-11T06:10:39.489 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on crush straw calc version zero = false 2021-02-11T06:10:39.489 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on no sortbitwise = false 2021-02-11T06:10:39.489 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on osd down out interval zero = false 2021-02-11T06:10:39.489 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on too few osds = false 2021-02-11T06:10:39.489 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-02-11T06:10:39.489 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_pool_no_redundancy = false 2021-02-11T06:10:39.489 INFO:teuthology.orchestra.run.smithi172.stdout: mon_allow_pool_size_one = true 2021-02-11T06:10:39.490 INFO:teuthology.orchestra.run.smithi172.stdout: 2021-02-11T06:10:39.490 INFO:teuthology.orchestra.run.smithi172.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-02-11T06:10:39.490 INFO:teuthology.orchestra.run.smithi172.stdout: 2021-02-11T06:10:39.490 INFO:teuthology.orchestra.run.smithi172.stdout: osd default data pool replay window = 5 2021-02-11T06:10:39.490 INFO:teuthology.orchestra.run.smithi172.stdout: 2021-02-11T06:10:39.490 INFO:teuthology.orchestra.run.smithi172.stdout: mon allow pool delete = true 2021-02-11T06:10:39.490 INFO:teuthology.orchestra.run.smithi172.stdout: 2021-02-11T06:10:39.490 INFO:teuthology.orchestra.run.smithi172.stdout: mon cluster log file level = debug 2021-02-11T06:10:39.490 INFO:teuthology.orchestra.run.smithi172.stdout: debug asserts on shutdown = true 2021-02-11T06:10:39.490 INFO:teuthology.orchestra.run.smithi172.stdout: mon health detail to clog = false 2021-02-11T06:10:39.490 INFO:teuthology.orchestra.run.smithi172.stdout: mon host = 172.21.15.61 2021-02-11T06:10:39.490 INFO:teuthology.orchestra.run.smithi172.stdout: osd heartbeat grace = 40 2021-02-11T06:10:39.490 INFO:teuthology.orchestra.run.smithi172.stdout: fsid = 043de53e-a645-454b-bce7-c489302846ce 2021-02-11T06:10:39.491 INFO:teuthology.orchestra.run.smithi172.stdout: 2021-02-11T06:10:39.491 INFO:teuthology.orchestra.run.smithi172.stdout:[osd] 2021-02-11T06:10:39.491 INFO:teuthology.orchestra.run.smithi172.stdout: osd journal size = 100 2021-02-11T06:10:39.491 INFO:teuthology.orchestra.run.smithi172.stdout: 2021-02-11T06:10:39.491 INFO:teuthology.orchestra.run.smithi172.stdout: osd scrub load threshold = 5.0 2021-02-11T06:10:39.491 INFO:teuthology.orchestra.run.smithi172.stdout: osd scrub max interval = 600 2021-02-11T06:10:39.491 INFO:teuthology.orchestra.run.smithi172.stdout: 2021-02-11T06:10:39.491 INFO:teuthology.orchestra.run.smithi172.stdout: osd recover clone overlap = true 2021-02-11T06:10:39.491 INFO:teuthology.orchestra.run.smithi172.stdout: osd recovery max chunk = 1048576 2021-02-11T06:10:39.491 INFO:teuthology.orchestra.run.smithi172.stdout: 2021-02-11T06:10:39.491 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug shutdown = true 2021-02-11T06:10:39.491 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug op order = true 2021-02-11T06:10:39.491 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug verify stray on activate = true 2021-02-11T06:10:39.491 INFO:teuthology.orchestra.run.smithi172.stdout: 2021-02-11T06:10:39.491 INFO:teuthology.orchestra.run.smithi172.stdout: osd open classes on start = true 2021-02-11T06:10:39.492 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug pg log writeout = true 2021-02-11T06:10:39.492 INFO:teuthology.orchestra.run.smithi172.stdout: 2021-02-11T06:10:39.492 INFO:teuthology.orchestra.run.smithi172.stdout: osd deep scrub update digest min age = 30 2021-02-11T06:10:39.492 INFO:teuthology.orchestra.run.smithi172.stdout: 2021-02-11T06:10:39.492 INFO:teuthology.orchestra.run.smithi172.stdout: osd map max advance = 10 2021-02-11T06:10:39.492 INFO:teuthology.orchestra.run.smithi172.stdout: 2021-02-11T06:10:39.492 INFO:teuthology.orchestra.run.smithi172.stdout: journal zero on create = true 2021-02-11T06:10:39.492 INFO:teuthology.orchestra.run.smithi172.stdout: 2021-02-11T06:10:39.492 INFO:teuthology.orchestra.run.smithi172.stdout: filestore ondisk finisher threads = 3 2021-02-11T06:10:39.492 INFO:teuthology.orchestra.run.smithi172.stdout: filestore apply finisher threads = 3 2021-02-11T06:10:39.492 INFO:teuthology.orchestra.run.smithi172.stdout: 2021-02-11T06:10:39.492 INFO:teuthology.orchestra.run.smithi172.stdout: bdev debug aio = true 2021-02-11T06:10:39.492 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug misdirected ops = true 2021-02-11T06:10:39.492 INFO:teuthology.orchestra.run.smithi172.stdout: debug filestore = 20 2021-02-11T06:10:39.493 INFO:teuthology.orchestra.run.smithi172.stdout: debug journal = 20 2021-02-11T06:10:39.493 INFO:teuthology.orchestra.run.smithi172.stdout: debug ms = 20 2021-02-11T06:10:39.493 INFO:teuthology.orchestra.run.smithi172.stdout: debug osd = 25 2021-02-11T06:10:39.493 INFO:teuthology.orchestra.run.smithi172.stdout: osd fast shutdown = False 2021-02-11T06:10:39.493 INFO:teuthology.orchestra.run.smithi172.stdout: rocksdb delete range threshold = 0 2021-02-11T06:10:39.493 INFO:teuthology.orchestra.run.smithi172.stdout: 2021-02-11T06:10:39.493 INFO:teuthology.orchestra.run.smithi172.stdout:[mgr] 2021-02-11T06:10:39.493 INFO:teuthology.orchestra.run.smithi172.stdout: debug ms = 1 2021-02-11T06:10:39.493 INFO:teuthology.orchestra.run.smithi172.stdout: debug mgr = 20 2021-02-11T06:10:39.493 INFO:teuthology.orchestra.run.smithi172.stdout: debug mon = 20 2021-02-11T06:10:39.493 INFO:teuthology.orchestra.run.smithi172.stdout: debug auth = 20 2021-02-11T06:10:39.493 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min pgs per osd = 4 2021-02-11T06:10:39.493 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min bytes per osd = 10 2021-02-11T06:10:39.493 INFO:teuthology.orchestra.run.smithi172.stdout: mgr/telemetry/nag = false 2021-02-11T06:10:39.493 INFO:teuthology.orchestra.run.smithi172.stdout: 2021-02-11T06:10:39.493 INFO:teuthology.orchestra.run.smithi172.stdout:[mon] 2021-02-11T06:10:39.494 INFO:teuthology.orchestra.run.smithi172.stdout: debug ms = 1 2021-02-11T06:10:39.494 INFO:teuthology.orchestra.run.smithi172.stdout: debug mon = 20 2021-02-11T06:10:39.494 INFO:teuthology.orchestra.run.smithi172.stdout: debug paxos = 20 2021-02-11T06:10:39.494 INFO:teuthology.orchestra.run.smithi172.stdout: debug auth = 20 2021-02-11T06:10:39.494 INFO:teuthology.orchestra.run.smithi172.stdout: mon data avail warn = 5 2021-02-11T06:10:39.494 INFO:teuthology.orchestra.run.smithi172.stdout: mon mgr mkfs grace = 240 2021-02-11T06:10:39.494 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min pgs per osd = 4 2021-02-11T06:10:39.494 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd reporter subtree level = osd 2021-02-11T06:10:39.494 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd prime pg temp = true 2021-02-11T06:10:39.494 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min bytes per osd = 10 2021-02-11T06:10:39.494 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd crush smoke test = False 2021-02-11T06:10:39.494 INFO:teuthology.orchestra.run.smithi172.stdout: 2021-02-11T06:10:39.494 INFO:teuthology.orchestra.run.smithi172.stdout:[client] 2021-02-11T06:10:39.494 INFO:teuthology.orchestra.run.smithi172.stdout: rgw cache enabled = true 2021-02-11T06:10:39.495 INFO:teuthology.orchestra.run.smithi172.stdout: rgw enable ops log = true 2021-02-11T06:10:39.495 INFO:teuthology.orchestra.run.smithi172.stdout: rgw enable usage log = true 2021-02-11T06:10:39.495 INFO:teuthology.orchestra.run.smithi172.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-02-11T06:10:39.495 INFO:teuthology.orchestra.run.smithi172.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-02-11T06:10:39.495 INFO:teuthology.orchestra.run.smithi172.stdout: debug rgw = 20 2021-02-11T06:10:39.495 INFO:teuthology.orchestra.run.smithi172.stdout: rgw crypt require ssl = False 2021-02-11T06:10:39.495 INFO:teuthology.orchestra.run.smithi172.stdout: rgw crypt s3 kms backend = testing 2021-02-11T06:10:39.495 INFO:teuthology.orchestra.run.smithi172.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-02-11T06:10:39.495 INFO:teuthology.orchestra.run.smithi172.stdout: rgw curl low speed time = 300 2021-02-11T06:10:39.495 INFO:teuthology.orchestra.run.smithi172.stdout: rgw data log num shards = 4 2021-02-11T06:10:39.495 INFO:teuthology.orchestra.run.smithi172.stdout: rgw md log max shards = 4 2021-02-11T06:10:39.495 INFO:teuthology.orchestra.run.smithi172.stdout: rgw sync log trim interval = 0 2021-02-11T06:10:39.495 INFO:teuthology.orchestra.run.smithi172.stdout: rgw sync obj etag verify = True 2021-02-11T06:10:39.495 INFO:teuthology.orchestra.run.smithi172.stdout: setgroup = ceph 2021-02-11T06:10:39.495 INFO:teuthology.orchestra.run.smithi172.stdout: setuser = ceph 2021-02-11T06:10:39.495 INFO:teuthology.orchestra.run.smithi172.stdout:[mon.a] 2021-02-11T06:10:39.496 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.0] 2021-02-11T06:10:39.496 INFO:teuthology.orchestra.run.smithi172.stdout: osd_max_omap_entries_per_request = 10 2021-02-11T06:10:39.496 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.1] 2021-02-11T06:10:39.496 INFO:teuthology.orchestra.run.smithi172.stdout: osd_max_omap_entries_per_request = 1000 2021-02-11T06:10:39.496 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.2] 2021-02-11T06:10:39.496 INFO:teuthology.orchestra.run.smithi172.stdout: osd_max_omap_entries_per_request = 10000 2021-02-11T06:10:39.497 INFO:teuthology.orchestra.run.smithi061.stdout:[global] 2021-02-11T06:10:39.497 INFO:teuthology.orchestra.run.smithi061.stdout: chdir = "" 2021-02-11T06:10:39.497 INFO:teuthology.orchestra.run.smithi061.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-02-11T06:10:39.497 INFO:teuthology.orchestra.run.smithi061.stdout: auth supported = cephx 2021-02-11T06:10:39.497 INFO:teuthology.orchestra.run.smithi061.stdout: 2021-02-11T06:10:39.497 INFO:teuthology.orchestra.run.smithi061.stdout: filestore xattr use omap = true 2021-02-11T06:10:39.497 INFO:teuthology.orchestra.run.smithi061.stdout: 2021-02-11T06:10:39.498 INFO:teuthology.orchestra.run.smithi061.stdout: mon clock drift allowed = 1.000 2021-02-11T06:10:39.498 INFO:teuthology.orchestra.run.smithi061.stdout: 2021-02-11T06:10:39.498 INFO:teuthology.orchestra.run.smithi061.stdout: osd crush chooseleaf type = 0 2021-02-11T06:10:39.498 INFO:teuthology.orchestra.run.smithi061.stdout: auth debug = true 2021-02-11T06:10:39.498 INFO:teuthology.orchestra.run.smithi061.stdout: 2021-02-11T06:10:39.498 INFO:teuthology.orchestra.run.smithi061.stdout: ms die on old message = true 2021-02-11T06:10:39.498 INFO:teuthology.orchestra.run.smithi061.stdout: ms die on bug = true 2021-02-11T06:10:39.498 INFO:teuthology.orchestra.run.smithi061.stdout: 2021-02-11T06:10:39.498 INFO:teuthology.orchestra.run.smithi061.stdout: mon max pg per osd = 10000 # >= luminous 2021-02-11T06:10:39.498 INFO:teuthology.orchestra.run.smithi061.stdout: mon pg warn max object skew = 0 2021-02-11T06:10:39.498 INFO:teuthology.orchestra.run.smithi061.stdout: 2021-02-11T06:10:39.498 INFO:teuthology.orchestra.run.smithi061.stdout: # disable pg_autoscaler by default for new pools 2021-02-11T06:10:39.498 INFO:teuthology.orchestra.run.smithi061.stdout: osd_pool_default_pg_autoscale_mode = off 2021-02-11T06:10:39.498 INFO:teuthology.orchestra.run.smithi061.stdout: 2021-02-11T06:10:39.499 INFO:teuthology.orchestra.run.smithi061.stdout: osd pool default size = 2 2021-02-11T06:10:39.499 INFO:teuthology.orchestra.run.smithi061.stdout: 2021-02-11T06:10:39.499 INFO:teuthology.orchestra.run.smithi061.stdout: mon osd allow primary affinity = true 2021-02-11T06:10:39.499 INFO:teuthology.orchestra.run.smithi061.stdout: mon osd allow pg remap = true 2021-02-11T06:10:39.499 INFO:teuthology.orchestra.run.smithi061.stdout: mon warn on legacy crush tunables = false 2021-02-11T06:10:39.499 INFO:teuthology.orchestra.run.smithi061.stdout: mon warn on crush straw calc version zero = false 2021-02-11T06:10:39.499 INFO:teuthology.orchestra.run.smithi061.stdout: mon warn on no sortbitwise = false 2021-02-11T06:10:39.499 INFO:teuthology.orchestra.run.smithi061.stdout: mon warn on osd down out interval zero = false 2021-02-11T06:10:39.499 INFO:teuthology.orchestra.run.smithi061.stdout: mon warn on too few osds = false 2021-02-11T06:10:39.499 INFO:teuthology.orchestra.run.smithi061.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-02-11T06:10:39.499 INFO:teuthology.orchestra.run.smithi061.stdout: mon_warn_on_pool_no_redundancy = false 2021-02-11T06:10:39.499 INFO:teuthology.orchestra.run.smithi061.stdout: mon_allow_pool_size_one = true 2021-02-11T06:10:39.499 INFO:teuthology.orchestra.run.smithi061.stdout: 2021-02-11T06:10:39.499 INFO:teuthology.orchestra.run.smithi061.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-02-11T06:10:39.499 INFO:teuthology.orchestra.run.smithi061.stdout: 2021-02-11T06:10:39.500 INFO:teuthology.orchestra.run.smithi061.stdout: osd default data pool replay window = 5 2021-02-11T06:10:39.500 INFO:teuthology.orchestra.run.smithi061.stdout: 2021-02-11T06:10:39.500 INFO:teuthology.orchestra.run.smithi061.stdout: mon allow pool delete = true 2021-02-11T06:10:39.500 INFO:teuthology.orchestra.run.smithi061.stdout: 2021-02-11T06:10:39.500 INFO:teuthology.orchestra.run.smithi061.stdout: mon cluster log file level = debug 2021-02-11T06:10:39.500 INFO:teuthology.orchestra.run.smithi061.stdout: debug asserts on shutdown = true 2021-02-11T06:10:39.500 INFO:teuthology.orchestra.run.smithi061.stdout: mon health detail to clog = false 2021-02-11T06:10:39.500 INFO:teuthology.orchestra.run.smithi061.stdout: mon host = 172.21.15.61 2021-02-11T06:10:39.500 INFO:teuthology.orchestra.run.smithi061.stdout: osd heartbeat grace = 40 2021-02-11T06:10:39.500 INFO:teuthology.orchestra.run.smithi061.stdout: fsid = 043de53e-a645-454b-bce7-c489302846ce 2021-02-11T06:10:39.500 INFO:teuthology.orchestra.run.smithi061.stdout: 2021-02-11T06:10:39.500 INFO:teuthology.orchestra.run.smithi061.stdout:[osd] 2021-02-11T06:10:39.500 INFO:teuthology.orchestra.run.smithi061.stdout: osd journal size = 100 2021-02-11T06:10:39.500 INFO:teuthology.orchestra.run.smithi061.stdout: 2021-02-11T06:10:39.501 INFO:teuthology.orchestra.run.smithi061.stdout: osd scrub load threshold = 5.0 2021-02-11T06:10:39.501 INFO:teuthology.orchestra.run.smithi061.stdout: osd scrub max interval = 600 2021-02-11T06:10:39.501 INFO:teuthology.orchestra.run.smithi061.stdout: 2021-02-11T06:10:39.501 INFO:teuthology.orchestra.run.smithi061.stdout: osd recover clone overlap = true 2021-02-11T06:10:39.501 INFO:teuthology.orchestra.run.smithi061.stdout: osd recovery max chunk = 1048576 2021-02-11T06:10:39.501 INFO:teuthology.orchestra.run.smithi061.stdout: 2021-02-11T06:10:39.501 INFO:teuthology.orchestra.run.smithi061.stdout: osd debug shutdown = true 2021-02-11T06:10:39.501 INFO:teuthology.orchestra.run.smithi061.stdout: osd debug op order = true 2021-02-11T06:10:39.501 INFO:teuthology.orchestra.run.smithi061.stdout: osd debug verify stray on activate = true 2021-02-11T06:10:39.501 INFO:teuthology.orchestra.run.smithi061.stdout: 2021-02-11T06:10:39.501 INFO:teuthology.orchestra.run.smithi061.stdout: osd open classes on start = true 2021-02-11T06:10:39.501 INFO:teuthology.orchestra.run.smithi061.stdout: osd debug pg log writeout = true 2021-02-11T06:10:39.501 INFO:teuthology.orchestra.run.smithi061.stdout: 2021-02-11T06:10:39.502 INFO:teuthology.orchestra.run.smithi061.stdout: osd deep scrub update digest min age = 30 2021-02-11T06:10:39.502 INFO:teuthology.orchestra.run.smithi061.stdout: 2021-02-11T06:10:39.502 INFO:teuthology.orchestra.run.smithi061.stdout: osd map max advance = 10 2021-02-11T06:10:39.502 INFO:teuthology.orchestra.run.smithi061.stdout: 2021-02-11T06:10:39.502 INFO:teuthology.orchestra.run.smithi061.stdout: journal zero on create = true 2021-02-11T06:10:39.502 INFO:teuthology.orchestra.run.smithi061.stdout: 2021-02-11T06:10:39.502 INFO:teuthology.orchestra.run.smithi061.stdout: filestore ondisk finisher threads = 3 2021-02-11T06:10:39.502 INFO:teuthology.orchestra.run.smithi061.stdout: filestore apply finisher threads = 3 2021-02-11T06:10:39.502 INFO:teuthology.orchestra.run.smithi061.stdout: 2021-02-11T06:10:39.502 INFO:teuthology.orchestra.run.smithi061.stdout: bdev debug aio = true 2021-02-11T06:10:39.502 INFO:teuthology.orchestra.run.smithi061.stdout: osd debug misdirected ops = true 2021-02-11T06:10:39.502 INFO:teuthology.orchestra.run.smithi061.stdout: debug filestore = 20 2021-02-11T06:10:39.502 INFO:teuthology.orchestra.run.smithi061.stdout: debug journal = 20 2021-02-11T06:10:39.502 INFO:teuthology.orchestra.run.smithi061.stdout: debug ms = 20 2021-02-11T06:10:39.503 INFO:teuthology.orchestra.run.smithi061.stdout: debug osd = 25 2021-02-11T06:10:39.503 INFO:teuthology.orchestra.run.smithi061.stdout: osd fast shutdown = False 2021-02-11T06:10:39.503 INFO:teuthology.orchestra.run.smithi061.stdout: rocksdb delete range threshold = 0 2021-02-11T06:10:39.503 INFO:teuthology.orchestra.run.smithi061.stdout: 2021-02-11T06:10:39.503 INFO:teuthology.orchestra.run.smithi061.stdout:[mgr] 2021-02-11T06:10:39.503 INFO:teuthology.orchestra.run.smithi061.stdout: debug ms = 1 2021-02-11T06:10:39.503 INFO:teuthology.orchestra.run.smithi061.stdout: debug mgr = 20 2021-02-11T06:10:39.503 INFO:teuthology.orchestra.run.smithi061.stdout: debug mon = 20 2021-02-11T06:10:39.503 INFO:teuthology.orchestra.run.smithi061.stdout: debug auth = 20 2021-02-11T06:10:39.503 INFO:teuthology.orchestra.run.smithi061.stdout: mon reweight min pgs per osd = 4 2021-02-11T06:10:39.503 INFO:teuthology.orchestra.run.smithi061.stdout: mon reweight min bytes per osd = 10 2021-02-11T06:10:39.503 INFO:teuthology.orchestra.run.smithi061.stdout: mgr/telemetry/nag = false 2021-02-11T06:10:39.503 INFO:teuthology.orchestra.run.smithi061.stdout: 2021-02-11T06:10:39.503 INFO:teuthology.orchestra.run.smithi061.stdout:[mon] 2021-02-11T06:10:39.503 INFO:teuthology.orchestra.run.smithi061.stdout: debug ms = 1 2021-02-11T06:10:39.504 INFO:teuthology.orchestra.run.smithi061.stdout: debug mon = 20 2021-02-11T06:10:39.504 INFO:teuthology.orchestra.run.smithi061.stdout: debug paxos = 20 2021-02-11T06:10:39.504 INFO:teuthology.orchestra.run.smithi061.stdout: debug auth = 20 2021-02-11T06:10:39.504 INFO:teuthology.orchestra.run.smithi061.stdout: mon data avail warn = 5 2021-02-11T06:10:39.504 INFO:teuthology.orchestra.run.smithi061.stdout: mon mgr mkfs grace = 240 2021-02-11T06:10:39.504 INFO:teuthology.orchestra.run.smithi061.stdout: mon reweight min pgs per osd = 4 2021-02-11T06:10:39.504 INFO:teuthology.orchestra.run.smithi061.stdout: mon osd reporter subtree level = osd 2021-02-11T06:10:39.504 INFO:teuthology.orchestra.run.smithi061.stdout: mon osd prime pg temp = true 2021-02-11T06:10:39.504 INFO:teuthology.orchestra.run.smithi061.stdout: mon reweight min bytes per osd = 10 2021-02-11T06:10:39.504 INFO:teuthology.orchestra.run.smithi061.stdout: mon osd crush smoke test = False 2021-02-11T06:10:39.504 INFO:teuthology.orchestra.run.smithi061.stdout: 2021-02-11T06:10:39.504 INFO:teuthology.orchestra.run.smithi061.stdout:[client] 2021-02-11T06:10:39.504 INFO:teuthology.orchestra.run.smithi061.stdout: rgw cache enabled = true 2021-02-11T06:10:39.504 INFO:teuthology.orchestra.run.smithi061.stdout: rgw enable ops log = true 2021-02-11T06:10:39.505 INFO:teuthology.orchestra.run.smithi061.stdout: rgw enable usage log = true 2021-02-11T06:10:39.505 INFO:teuthology.orchestra.run.smithi061.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-02-11T06:10:39.505 INFO:teuthology.orchestra.run.smithi061.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-02-11T06:10:39.505 INFO:teuthology.orchestra.run.smithi061.stdout: debug rgw = 20 2021-02-11T06:10:39.505 INFO:teuthology.orchestra.run.smithi061.stdout: rgw crypt require ssl = False 2021-02-11T06:10:39.505 INFO:teuthology.orchestra.run.smithi061.stdout: rgw crypt s3 kms backend = testing 2021-02-11T06:10:39.505 INFO:teuthology.orchestra.run.smithi061.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-02-11T06:10:39.505 INFO:teuthology.orchestra.run.smithi061.stdout: rgw curl low speed time = 300 2021-02-11T06:10:39.505 INFO:teuthology.orchestra.run.smithi061.stdout: rgw data log num shards = 4 2021-02-11T06:10:39.505 INFO:teuthology.orchestra.run.smithi061.stdout: rgw md log max shards = 4 2021-02-11T06:10:39.505 INFO:teuthology.orchestra.run.smithi061.stdout: rgw sync log trim interval = 0 2021-02-11T06:10:39.505 INFO:teuthology.orchestra.run.smithi061.stdout: rgw sync obj etag verify = True 2021-02-11T06:10:39.505 INFO:teuthology.orchestra.run.smithi061.stdout: setgroup = ceph 2021-02-11T06:10:39.505 INFO:teuthology.orchestra.run.smithi061.stdout: setuser = ceph 2021-02-11T06:10:39.506 INFO:teuthology.orchestra.run.smithi061.stdout:[mon.a] 2021-02-11T06:10:39.506 INFO:teuthology.orchestra.run.smithi061.stdout:[osd.0] 2021-02-11T06:10:39.506 INFO:teuthology.orchestra.run.smithi061.stdout: osd_max_omap_entries_per_request = 10 2021-02-11T06:10:39.506 INFO:teuthology.orchestra.run.smithi061.stdout:[osd.1] 2021-02-11T06:10:39.506 INFO:teuthology.orchestra.run.smithi061.stdout: osd_max_omap_entries_per_request = 1000 2021-02-11T06:10:39.506 INFO:teuthology.orchestra.run.smithi061.stdout:[osd.2] 2021-02-11T06:10:39.506 INFO:teuthology.orchestra.run.smithi061.stdout: osd_max_omap_entries_per_request = 10000 2021-02-11T06:10:39.506 INFO:tasks.ceph:Creating admin key on c1.mon.a... 2021-02-11T06:10:39.506 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c1.keyring 2021-02-11T06:10:39.586 INFO:tasks.ceph:Copying monmap to all nodes... 2021-02-11T06:10:39.586 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2021-02-11T06:10:39.586 DEBUG:teuthology.orchestra.run.smithi061:> dd if=/etc/ceph/c1.keyring of=/dev/stdout 2021-02-11T06:10:39.633 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2021-02-11T06:10:39.633 DEBUG:teuthology.orchestra.run.smithi061:> dd if=/home/ubuntu/cephtest/c1.monmap of=/dev/stdout 2021-02-11T06:10:39.681 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi061.front.sepia.ceph.com 2021-02-11T06:10:39.681 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2021-02-11T06:10:39.681 DEBUG:teuthology.orchestra.run.smithi061:> sudo dd of=/etc/ceph/c1.keyring 2021-02-11T06:10:39.681 DEBUG:teuthology.orchestra.run.smithi061:> sudo chmod 0644 /etc/ceph/c1.keyring 2021-02-11T06:10:39.748 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2021-02-11T06:10:39.748 DEBUG:teuthology.orchestra.run.smithi061:> dd of=/home/ubuntu/cephtest/c1.monmap 2021-02-11T06:10:39.797 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi172.front.sepia.ceph.com 2021-02-11T06:10:39.797 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2021-02-11T06:10:39.797 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/ceph/c1.keyring 2021-02-11T06:10:39.797 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 0644 /etc/ceph/c1.keyring 2021-02-11T06:10:39.822 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2021-02-11T06:10:39.822 DEBUG:teuthology.orchestra.run.smithi172:> dd of=/home/ubuntu/cephtest/c1.monmap 2021-02-11T06:10:39.873 INFO:tasks.ceph:Setting up mon nodes... 2021-02-11T06:10:39.873 INFO:tasks.ceph:Setting up mgr nodes... 2021-02-11T06:10:39.873 DEBUG:teuthology.orchestra.run.smithi061:> sudo mkdir -p /var/lib/ceph/mgr/c1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c1-x/keyring 2021-02-11T06:10:39.917 INFO:teuthology.orchestra.run.smithi061.stdout:creating /var/lib/ceph/mgr/c1-x/keyring 2021-02-11T06:10:39.919 INFO:tasks.ceph:Setting up mds nodes... 2021-02-11T06:10:39.919 INFO:tasks.ceph_client:Setting up client nodes... 2021-02-11T06:10:39.919 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c1.client.0.keyring && sudo chmod 0644 /etc/ceph/c1.client.0.keyring 2021-02-11T06:10:39.996 INFO:teuthology.orchestra.run.smithi061.stdout:creating /etc/ceph/c1.client.0.keyring 2021-02-11T06:10:40.009 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c1.client.1.keyring && sudo chmod 0644 /etc/ceph/c1.client.1.keyring 2021-02-11T06:10:40.084 INFO:teuthology.orchestra.run.smithi061.stdout:creating /etc/ceph/c1.client.1.keyring 2021-02-11T06:10:40.096 INFO:tasks.ceph:Running mkfs on osd nodes... 2021-02-11T06:10:40.096 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi061.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2021-02-11T06:10:40.097 DEBUG:teuthology.orchestra.run.smithi061:> sudo mkdir -p /var/lib/ceph/osd/c1-0 2021-02-11T06:10:40.154 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2021-02-11T06:10:40.154 INFO:tasks.ceph:role: c1.osd.0 2021-02-11T06:10:40.154 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi061.front.sepia.ceph.com 2021-02-11T06:10:40.154 DEBUG:teuthology.orchestra.run.smithi061:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2021-02-11T06:10:40.674 INFO:teuthology.orchestra.run.smithi061.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2021-02-11T06:10:40.675 INFO:teuthology.orchestra.run.smithi061.stdout: = sectsz=512 attr=2, projid32bit=1 2021-02-11T06:10:40.675 INFO:teuthology.orchestra.run.smithi061.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2021-02-11T06:10:40.675 INFO:teuthology.orchestra.run.smithi061.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-02-11T06:10:40.675 INFO:teuthology.orchestra.run.smithi061.stdout: = sunit=0 swidth=0 blks 2021-02-11T06:10:40.675 INFO:teuthology.orchestra.run.smithi061.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-02-11T06:10:40.675 INFO:teuthology.orchestra.run.smithi061.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-02-11T06:10:40.675 INFO:teuthology.orchestra.run.smithi061.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-02-11T06:10:40.675 INFO:teuthology.orchestra.run.smithi061.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-02-11T06:10:40.676 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi061.front.sepia.ceph.com -o noatime 2021-02-11T06:10:40.676 DEBUG:teuthology.orchestra.run.smithi061:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c1-0 2021-02-11T06:10:40.750 DEBUG:teuthology.orchestra.run.smithi061:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-0 2021-02-11T06:10:40.761 INFO:teuthology.orchestra.run.smithi061.stderr:sudo: /sbin/restorecon: command not found 2021-02-11T06:10:40.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-11T06:10:40.762 DEBUG:teuthology.orchestra.run.smithi061:> sudo mkdir -p /var/lib/ceph/osd/c1-1 2021-02-11T06:10:40.817 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2021-02-11T06:10:40.817 INFO:tasks.ceph:role: c1.osd.1 2021-02-11T06:10:40.817 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi061.front.sepia.ceph.com 2021-02-11T06:10:40.818 DEBUG:teuthology.orchestra.run.smithi061:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2021-02-11T06:10:41.332 INFO:teuthology.orchestra.run.smithi061.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2021-02-11T06:10:41.333 INFO:teuthology.orchestra.run.smithi061.stdout: = sectsz=512 attr=2, projid32bit=1 2021-02-11T06:10:41.333 INFO:teuthology.orchestra.run.smithi061.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2021-02-11T06:10:41.333 INFO:teuthology.orchestra.run.smithi061.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-02-11T06:10:41.333 INFO:teuthology.orchestra.run.smithi061.stdout: = sunit=0 swidth=0 blks 2021-02-11T06:10:41.333 INFO:teuthology.orchestra.run.smithi061.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-02-11T06:10:41.333 INFO:teuthology.orchestra.run.smithi061.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-02-11T06:10:41.333 INFO:teuthology.orchestra.run.smithi061.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-02-11T06:10:41.333 INFO:teuthology.orchestra.run.smithi061.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-02-11T06:10:41.334 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi061.front.sepia.ceph.com -o noatime 2021-02-11T06:10:41.334 DEBUG:teuthology.orchestra.run.smithi061:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c1-1 2021-02-11T06:10:41.407 DEBUG:teuthology.orchestra.run.smithi061:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-1 2021-02-11T06:10:41.419 INFO:teuthology.orchestra.run.smithi061.stderr:sudo: /sbin/restorecon: command not found 2021-02-11T06:10:41.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-11T06:10:41.420 DEBUG:teuthology.orchestra.run.smithi061:> sudo mkdir -p /var/lib/ceph/osd/c1-2 2021-02-11T06:10:41.478 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2021-02-11T06:10:41.478 INFO:tasks.ceph:role: c1.osd.2 2021-02-11T06:10:41.478 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi061.front.sepia.ceph.com 2021-02-11T06:10:41.479 DEBUG:teuthology.orchestra.run.smithi061:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2021-02-11T06:10:41.976 INFO:teuthology.orchestra.run.smithi061.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2021-02-11T06:10:41.976 INFO:teuthology.orchestra.run.smithi061.stdout: = sectsz=512 attr=2, projid32bit=1 2021-02-11T06:10:41.976 INFO:teuthology.orchestra.run.smithi061.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2021-02-11T06:10:41.977 INFO:teuthology.orchestra.run.smithi061.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-02-11T06:10:41.977 INFO:teuthology.orchestra.run.smithi061.stdout: = sunit=0 swidth=0 blks 2021-02-11T06:10:41.977 INFO:teuthology.orchestra.run.smithi061.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-02-11T06:10:41.977 INFO:teuthology.orchestra.run.smithi061.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-02-11T06:10:41.977 INFO:teuthology.orchestra.run.smithi061.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-02-11T06:10:41.977 INFO:teuthology.orchestra.run.smithi061.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-02-11T06:10:41.978 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi061.front.sepia.ceph.com -o noatime 2021-02-11T06:10:41.978 DEBUG:teuthology.orchestra.run.smithi061:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c1-2 2021-02-11T06:10:42.034 DEBUG:teuthology.orchestra.run.smithi061:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-2 2021-02-11T06:10:43.715 INFO:teuthology.orchestra.run.smithi061.stderr:sudo: /sbin/restorecon: command not found 2021-02-11T06:10:43.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-11T06:10:43.716 DEBUG:teuthology.orchestra.run.smithi061:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c1.monmap 2021-02-11T06:10:43.801 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:10:43.800+0000 7fa477aae080 -1 auth: error reading file: /var/lib/ceph/osd/c1-0/keyring: can't open /var/lib/ceph/osd/c1-0/keyring: (2) No such file or directory 2021-02-11T06:10:43.802 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:10:43.800+0000 7fa477aae080 -1 created new key in keyring /var/lib/ceph/osd/c1-0/keyring 2021-02-11T06:10:43.802 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:10:43.800+0000 7fa477aae080 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2021-02-11T06:10:43.802 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:10:43.800+0000 7fa477aae080 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2021-02-11T06:10:43.802 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:10:43.800+0000 7fa477aae080 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2021-02-11T06:10:43.803 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:10:43.800+0000 7fa477aae080 -1 bluestore(/var/lib/ceph/osd/c1-0) _read_fsid unparsable uuid 2021-02-11T06:10:45.992 DEBUG:teuthology.orchestra.run.smithi061:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-0 2021-02-11T06:10:46.049 DEBUG:teuthology.orchestra.run.smithi061:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c1.monmap 2021-02-11T06:10:46.130 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:10:46.128+0000 7fecf4648080 -1 auth: error reading file: /var/lib/ceph/osd/c1-1/keyring: can't open /var/lib/ceph/osd/c1-1/keyring: (2) No such file or directory 2021-02-11T06:10:46.130 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:10:46.128+0000 7fecf4648080 -1 created new key in keyring /var/lib/ceph/osd/c1-1/keyring 2021-02-11T06:10:46.130 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:10:46.128+0000 7fecf4648080 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2021-02-11T06:10:46.130 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:10:46.128+0000 7fecf4648080 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2021-02-11T06:10:46.130 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:10:46.128+0000 7fecf4648080 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2021-02-11T06:10:46.131 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:10:46.128+0000 7fecf4648080 -1 bluestore(/var/lib/ceph/osd/c1-1) _read_fsid unparsable uuid 2021-02-11T06:10:48.311 DEBUG:teuthology.orchestra.run.smithi061:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-1 2021-02-11T06:10:48.371 DEBUG:teuthology.orchestra.run.smithi061:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c1.monmap 2021-02-11T06:10:48.455 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:10:48.452+0000 7f233493d080 -1 auth: error reading file: /var/lib/ceph/osd/c1-2/keyring: can't open /var/lib/ceph/osd/c1-2/keyring: (2) No such file or directory 2021-02-11T06:10:48.456 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:10:48.452+0000 7f233493d080 -1 created new key in keyring /var/lib/ceph/osd/c1-2/keyring 2021-02-11T06:10:48.456 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:10:48.452+0000 7f233493d080 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2021-02-11T06:10:48.456 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:10:48.456+0000 7f233493d080 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2021-02-11T06:10:48.456 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:10:48.456+0000 7f233493d080 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2021-02-11T06:10:48.456 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:10:48.456+0000 7f233493d080 -1 bluestore(/var/lib/ceph/osd/c1-2) _read_fsid unparsable uuid 2021-02-11T06:10:50.647 DEBUG:teuthology.orchestra.run.smithi061:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-2 2021-02-11T06:10:50.703 INFO:tasks.ceph:Reading keys from all nodes... 2021-02-11T06:10:50.703 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2021-02-11T06:10:50.703 DEBUG:teuthology.orchestra.run.smithi061:> sudo dd if=/var/lib/ceph/mgr/c1-x/keyring of=/dev/stdout 2021-02-11T06:10:50.762 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2021-02-11T06:10:50.762 DEBUG:teuthology.orchestra.run.smithi061:> sudo dd if=/var/lib/ceph/osd/c1-0/keyring of=/dev/stdout 2021-02-11T06:10:50.817 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2021-02-11T06:10:50.817 DEBUG:teuthology.orchestra.run.smithi061:> sudo dd if=/var/lib/ceph/osd/c1-1/keyring of=/dev/stdout 2021-02-11T06:10:50.873 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2021-02-11T06:10:50.874 DEBUG:teuthology.orchestra.run.smithi061:> sudo dd if=/var/lib/ceph/osd/c1-2/keyring of=/dev/stdout 2021-02-11T06:10:50.929 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2021-02-11T06:10:50.929 DEBUG:teuthology.orchestra.run.smithi061:> dd if=/etc/ceph/c1.client.0.keyring of=/dev/stdout 2021-02-11T06:10:50.977 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2021-02-11T06:10:50.977 DEBUG:teuthology.orchestra.run.smithi061:> dd if=/etc/ceph/c1.client.1.keyring of=/dev/stdout 2021-02-11T06:10:51.025 INFO:tasks.ceph:Adding keys to all mons... 2021-02-11T06:10:51.025 DEBUG:teuthology.orchestra.run.smithi061:> sudo tee -a /etc/ceph/c1.keyring 2021-02-11T06:10:51.080 INFO:teuthology.orchestra.run.smithi061.stdout:[mgr.x] 2021-02-11T06:10:51.080 INFO:teuthology.orchestra.run.smithi061.stdout: key = AQBfyiRgM17NNhAA+Gp67JxCsEmgFRdZplz7Kw== 2021-02-11T06:10:51.080 INFO:teuthology.orchestra.run.smithi061.stdout:[osd.0] 2021-02-11T06:10:51.080 INFO:teuthology.orchestra.run.smithi061.stdout: key = AQBjyiRgOW7nLxAA9B6X9n8LzbzdtBheoHWC/A== 2021-02-11T06:10:51.080 INFO:teuthology.orchestra.run.smithi061.stdout:[osd.1] 2021-02-11T06:10:51.081 INFO:teuthology.orchestra.run.smithi061.stdout: key = AQBmyiRg/ALbBxAAVGKFTp4Fo6YIKjLJqwBSjw== 2021-02-11T06:10:51.081 INFO:teuthology.orchestra.run.smithi061.stdout:[osd.2] 2021-02-11T06:10:51.081 INFO:teuthology.orchestra.run.smithi061.stdout: key = AQBoyiRgkVZEGxAACrL6XGtJDROjI3xnHF2+4g== 2021-02-11T06:10:51.081 INFO:teuthology.orchestra.run.smithi061.stdout:[client.0] 2021-02-11T06:10:51.081 INFO:teuthology.orchestra.run.smithi061.stdout: key = AQBfyiRgF+WGOxAApEzgrhvXfo/qcYJ+M+4Xkw== 2021-02-11T06:10:51.081 INFO:teuthology.orchestra.run.smithi061.stdout:[client.1] 2021-02-11T06:10:51.081 INFO:teuthology.orchestra.run.smithi061.stdout: key = AQBgyiRghk8sBRAAwJn51ie8Hh0Qqx4CLtEp2w== 2021-02-11T06:10:51.082 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2021-02-11T06:10:51.166 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-11T06:10:51.249 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-11T06:10:51.334 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-11T06:10:51.416 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-02-11T06:10:51.504 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-02-11T06:10:51.544 INFO:tasks.ceph:Running mkfs on mon nodes... 2021-02-11T06:10:51.545 DEBUG:teuthology.orchestra.run.smithi061:> sudo mkdir -p /var/lib/ceph/mon/c1-a 2021-02-11T06:10:51.602 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c1 --mkfs -i a --monmap /home/ubuntu/cephtest/c1.monmap --keyring /etc/ceph/c1.keyring 2021-02-11T06:10:51.696 DEBUG:teuthology.orchestra.run.smithi061:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c1-a 2021-02-11T06:10:51.711 DEBUG:teuthology.orchestra.run.smithi061:> rm -- /home/ubuntu/cephtest/c1.monmap 2021-02-11T06:10:51.761 INFO:tasks.ceph:Starting mon daemons in cluster c1... 2021-02-11T06:10:51.761 INFO:tasks.ceph.c1.mon.a:Restarting daemon 2021-02-11T06:10:51.761 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c1 -i a 2021-02-11T06:10:51.804 INFO:tasks.ceph.c1.mon.a:Started 2021-02-11T06:10:51.805 INFO:tasks.ceph:Starting mgr daemons in cluster c1... 2021-02-11T06:10:51.805 INFO:tasks.ceph.c1.mgr.x:Restarting daemon 2021-02-11T06:10:51.805 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c1 -i x 2021-02-11T06:10:51.807 INFO:tasks.ceph.c1.mgr.x:Started 2021-02-11T06:10:51.807 INFO:tasks.ceph:Setting crush tunables to default 2021-02-11T06:10:51.807 DEBUG:teuthology.orchestra.run.smithi061:> sudo ceph --cluster c1 osd crush tunables default 2021-02-11T06:10:52.182 INFO:teuthology.orchestra.run.smithi061.stderr:adjusted tunables profile to default 2021-02-11T06:10:52.193 INFO:tasks.ceph:Starting osd daemons in cluster c1... 2021-02-11T06:10:52.193 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2021-02-11T06:10:52.193 DEBUG:teuthology.orchestra.run.smithi061:> sudo dd if=/var/lib/ceph/osd/c1-0/fsid of=/dev/stdout 2021-02-11T06:10:52.206 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2021-02-11T06:10:52.206 DEBUG:teuthology.orchestra.run.smithi061:> sudo dd if=/var/lib/ceph/osd/c1-1/fsid of=/dev/stdout 2021-02-11T06:10:52.260 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2021-02-11T06:10:52.260 DEBUG:teuthology.orchestra.run.smithi061:> sudo dd if=/var/lib/ceph/osd/c1-2/fsid of=/dev/stdout 2021-02-11T06:10:52.317 DEBUG:teuthology.orchestra.run.smithi061:> sudo ceph --cluster c1 osd new 0d97d5d6-5f78-4266-9bb1-8f868bd62e54 0 2021-02-11T06:10:52.583 INFO:teuthology.orchestra.run.smithi061.stdout:0 2021-02-11T06:10:52.601 DEBUG:teuthology.orchestra.run.smithi061:> sudo ceph --cluster c1 osd new 2b921cd0-ae04-4e00-abee-e6327aaff01f 1 2021-02-11T06:10:52.815 INFO:teuthology.orchestra.run.smithi061.stdout:1 2021-02-11T06:10:52.829 DEBUG:teuthology.orchestra.run.smithi061:> sudo ceph --cluster c1 osd new 1d1b38e0-b970-4074-9525-26fef2f47dad 2 2021-02-11T06:10:53.042 INFO:teuthology.orchestra.run.smithi061.stdout:2 2021-02-11T06:10:53.055 INFO:tasks.ceph.c1.osd.0:Restarting daemon 2021-02-11T06:10:53.055 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 0 2021-02-11T06:10:53.057 INFO:tasks.ceph.c1.osd.0:Started 2021-02-11T06:10:53.057 INFO:tasks.ceph.c1.osd.1:Restarting daemon 2021-02-11T06:10:53.058 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 1 2021-02-11T06:10:53.059 INFO:tasks.ceph.c1.osd.1:Started 2021-02-11T06:10:53.060 INFO:tasks.ceph.c1.osd.2:Restarting daemon 2021-02-11T06:10:53.060 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 2 2021-02-11T06:10:53.061 INFO:tasks.ceph.c1.osd.2:Started 2021-02-11T06:10:53.062 INFO:tasks.ceph:Waiting for OSDs to come up 2021-02-11T06:10:53.365 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2021-02-11T06:11:06.661 INFO:tasks.ceph.c1.osd.1.smithi061.stderr:2021-02-11T06:10:53.412+0000 7f2d63ebd080 -1 Falling back to public interface 2021-02-11T06:11:06.661 INFO:tasks.ceph.c1.osd.1.smithi061.stderr:2021-02-11T06:10:56.116+0000 7f2d63ebd080 -1 osd.1 0 log_to_monitors {default=true} 2021-02-11T06:11:06.661 INFO:tasks.ceph.c1.osd.1.smithi061.stderr:2021-02-11T06:10:57.188+0000 7f2d5b386700 -1 osd.1 0 waiting for initial osdmap 2021-02-11T06:11:06.661 INFO:tasks.ceph.c1.osd.0.smithi061.stderr:2021-02-11T06:10:53.656+0000 7f75c084d080 -1 Falling back to public interface 2021-02-11T06:11:06.661 INFO:tasks.ceph.c1.osd.0.smithi061.stderr:2021-02-11T06:10:56.648+0000 7f75c084d080 -1 osd.0 0 log_to_monitors {default=true} 2021-02-11T06:11:06.661 INFO:tasks.ceph.c1.osd.0.smithi061.stderr:2021-02-11T06:10:58.204+0000 7f75b9519700 -1 osd.0 0 waiting for initial osdmap 2021-02-11T06:11:06.662 INFO:tasks.ceph.c1.osd.2.smithi061.stderr:2021-02-11T06:10:53.672+0000 7fceeae27080 -1 Falling back to public interface 2021-02-11T06:11:06.662 INFO:tasks.ceph.c1.osd.2.smithi061.stderr:2021-02-11T06:10:56.696+0000 7fceeae27080 -1 osd.2 0 log_to_monitors {default=true} 2021-02-11T06:11:06.662 INFO:tasks.ceph.c1.osd.2.smithi061.stderr:2021-02-11T06:10:58.204+0000 7fcee22f0700 -1 osd.2 0 waiting for initial osdmap 2021-02-11T06:11:06.963 INFO:teuthology.misc.health.smithi061.stdout: 2021-02-11T06:11:06.963 INFO:teuthology.misc.health.smithi061.stdout:{"epoch":13,"fsid":"043de53e-a645-454b-bce7-c489302846ce","created":"2021-02-11T06:10:51.924984+0000","modified":"2021-02-11T06:11:03.204882+0000","last_up_change":"2021-02-11T06:10:59.194103+0000","last_in_change":"2021-02-11T06:10:59.194103+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-02-11T06:10:59.208467+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"0d97d5d6-5f78-4266-9bb1-8f868bd62e54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":14659},{"type":"v1","addr":"172.21.15.61:6809","nonce":14659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":14659},{"type":"v1","addr":"172.21.15.61:6811","nonce":14659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":14659},{"type":"v1","addr":"172.21.15.61:6815","nonce":14659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":14659},{"type":"v1","addr":"172.21.15.61:6813","nonce":14659}]},"public_addr":"172.21.15.61:6809/14659","cluster_addr":"172.21.15.61:6811/14659","heartbeat_back_addr":"172.21.15.61:6815/14659","heartbeat_front_addr":"172.21.15.61:6813/14659","state":["exists","up"]},{"osd":1,"uuid":"2b921cd0-ae04-4e00-abee-e6327aaff01f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":14661},{"type":"v1","addr":"172.21.15.61:6801","nonce":14661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":14661},{"type":"v1","addr":"172.21.15.61:6803","nonce":14661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":14661},{"type":"v1","addr":"172.21.15.61:6807","nonce":14661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":14661},{"type":"v1","addr":"172.21.15.61:6805","nonce":14661}]},"public_addr":"172.21.15.61:6801/14661","cluster_addr":"172.21.15.61:6803/14661","heartbeat_back_addr":"172.21.15.61:6807/14661","heartbeat_front_addr":"172.21.15.61:6805/14661","state":["exists","up"]},{"osd":2,"uuid":"1d1b38e0-b970-4074-9525-26fef2f47dad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":14660},{"type":"v1","addr":"172.21.15.61:6817","nonce":14660}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":14660},{"type":"v1","addr":"172.21.15.61:6819","nonce":14660}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":14660},{"type":"v1","addr":"172.21.15.61:6823","nonce":14660}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":14660},{"type":"v1","addr":"172.21.15.61:6821","nonce":14660}]},"public_addr":"172.21.15.61:6817/14660","cluster_addr":"172.21.15.61:6819/14660","heartbeat_back_addr":"172.21.15.61:6823/14660","heartbeat_front_addr":"172.21.15.61:6821/14660","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T06:10:57.694212+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T06:10:57.074765+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T06:10:57.729633+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-02-11T06:11:06.982 DEBUG:teuthology.misc:3 of 3 OSDs are up 2021-02-11T06:11:06.982 INFO:tasks.ceph:Creating RBD pool 2021-02-11T06:11:06.982 DEBUG:teuthology.orchestra.run.smithi061:> sudo ceph --cluster c1 osd pool create rbd 8 2021-02-11T06:11:07.914 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'rbd' created 2021-02-11T06:11:07.928 DEBUG:teuthology.orchestra.run.smithi061:> sudo ceph --cluster c1 osd pool application enable rbd rbd --yes-i-really-mean-it 2021-02-11T06:11:08.920 INFO:teuthology.orchestra.run.smithi061.stderr:enabled application 'rbd' on pool 'rbd' 2021-02-11T06:11:08.933 INFO:tasks.ceph:Starting mds daemons in cluster c1... 2021-02-11T06:11:08.933 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2021-02-11T06:11:08.934 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2021-02-11T06:11:09.230 INFO:teuthology.orchestra.run.smithi061.stdout: 2021-02-11T06:11:09.230 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":15,"fsid":"043de53e-a645-454b-bce7-c489302846ce","created":"2021-02-11T06:10:51.924984+0000","modified":"2021-02-11T06:11:08.901179+0000","last_up_change":"2021-02-11T06:10:59.194103+0000","last_in_change":"2021-02-11T06:10:59.194103+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-02-11T06:10:59.208467+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-02-11T06:11:07.282762+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"0d97d5d6-5f78-4266-9bb1-8f868bd62e54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":14659},{"type":"v1","addr":"172.21.15.61:6809","nonce":14659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":14659},{"type":"v1","addr":"172.21.15.61:6811","nonce":14659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":14659},{"type":"v1","addr":"172.21.15.61:6815","nonce":14659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":14659},{"type":"v1","addr":"172.21.15.61:6813","nonce":14659}]},"public_addr":"172.21.15.61:6809/14659","cluster_addr":"172.21.15.61:6811/14659","heartbeat_back_addr":"172.21.15.61:6815/14659","heartbeat_front_addr":"172.21.15.61:6813/14659","state":["exists","up"]},{"osd":1,"uuid":"2b921cd0-ae04-4e00-abee-e6327aaff01f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":14661},{"type":"v1","addr":"172.21.15.61:6801","nonce":14661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":14661},{"type":"v1","addr":"172.21.15.61:6803","nonce":14661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":14661},{"type":"v1","addr":"172.21.15.61:6807","nonce":14661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":14661},{"type":"v1","addr":"172.21.15.61:6805","nonce":14661}]},"public_addr":"172.21.15.61:6801/14661","cluster_addr":"172.21.15.61:6803/14661","heartbeat_back_addr":"172.21.15.61:6807/14661","heartbeat_front_addr":"172.21.15.61:6805/14661","state":["exists","up"]},{"osd":2,"uuid":"1d1b38e0-b970-4074-9525-26fef2f47dad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":14660},{"type":"v1","addr":"172.21.15.61:6817","nonce":14660}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":14660},{"type":"v1","addr":"172.21.15.61:6819","nonce":14660}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":14660},{"type":"v1","addr":"172.21.15.61:6823","nonce":14660}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":14660},{"type":"v1","addr":"172.21.15.61:6821","nonce":14660}]},"public_addr":"172.21.15.61:6817/14660","cluster_addr":"172.21.15.61:6819/14660","heartbeat_back_addr":"172.21.15.61:6823/14660","heartbeat_front_addr":"172.21.15.61:6821/14660","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T06:10:57.694212+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T06:10:57.074765+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T06:10:57.729633+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-02-11T06:11:09.244 INFO:tasks.ceph.ceph_manager.c1:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2021-02-11T06:10:59.208467+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '13', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2021-02-11T06:11:07.282762+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-11T06:11:09.245 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd pool get device_health_metrics pg_num 2021-02-11T06:11:09.558 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2021-02-11T06:11:09.573 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd pool get rbd pg_num 2021-02-11T06:11:09.866 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2021-02-11T06:11:09.881 INFO:tasks.ceph:Waiting until c1 daemons up and pgs clean... 2021-02-11T06:11:09.881 INFO:tasks.ceph.ceph_manager.c1:waiting for mgr available 2021-02-11T06:11:09.881 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 mgr dump --format=json 2021-02-11T06:11:10.219 INFO:teuthology.orchestra.run.smithi061.stdout: 2021-02-11T06:11:10.234 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":4,"active_gid":4099,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":14476},{"type":"v1","addr":"172.21.15.61:6825","nonce":14476}]},"active_addr":"172.21.15.61:6825/14476","active_change":"2021-02-11T06:10:56.906918+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading remoto library:No module named 'remoto'","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"haproxy","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.61:0","nonce":1231112972}]},{"addrvec":[{"type":"v2","addr":"172.21.15.61:0","nonce":876426496}]}]}} 2021-02-11T06:11:10.236 INFO:tasks.ceph.ceph_manager.c1:mgr available! 2021-02-11T06:11:10.236 INFO:tasks.ceph.ceph_manager.c1:waiting for all up 2021-02-11T06:11:10.236 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2021-02-11T06:11:10.532 INFO:teuthology.orchestra.run.smithi061.stdout: 2021-02-11T06:11:10.533 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":16,"fsid":"043de53e-a645-454b-bce7-c489302846ce","created":"2021-02-11T06:10:51.924984+0000","modified":"2021-02-11T06:11:09.909847+0000","last_up_change":"2021-02-11T06:10:59.194103+0000","last_in_change":"2021-02-11T06:10:59.194103+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-02-11T06:10:59.208467+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-02-11T06:11:07.282762+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"0d97d5d6-5f78-4266-9bb1-8f868bd62e54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":14659},{"type":"v1","addr":"172.21.15.61:6809","nonce":14659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":14659},{"type":"v1","addr":"172.21.15.61:6811","nonce":14659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":14659},{"type":"v1","addr":"172.21.15.61:6815","nonce":14659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":14659},{"type":"v1","addr":"172.21.15.61:6813","nonce":14659}]},"public_addr":"172.21.15.61:6809/14659","cluster_addr":"172.21.15.61:6811/14659","heartbeat_back_addr":"172.21.15.61:6815/14659","heartbeat_front_addr":"172.21.15.61:6813/14659","state":["exists","up"]},{"osd":1,"uuid":"2b921cd0-ae04-4e00-abee-e6327aaff01f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":14661},{"type":"v1","addr":"172.21.15.61:6801","nonce":14661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":14661},{"type":"v1","addr":"172.21.15.61:6803","nonce":14661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":14661},{"type":"v1","addr":"172.21.15.61:6807","nonce":14661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":14661},{"type":"v1","addr":"172.21.15.61:6805","nonce":14661}]},"public_addr":"172.21.15.61:6801/14661","cluster_addr":"172.21.15.61:6803/14661","heartbeat_back_addr":"172.21.15.61:6807/14661","heartbeat_front_addr":"172.21.15.61:6805/14661","state":["exists","up"]},{"osd":2,"uuid":"1d1b38e0-b970-4074-9525-26fef2f47dad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":14660},{"type":"v1","addr":"172.21.15.61:6817","nonce":14660}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":14660},{"type":"v1","addr":"172.21.15.61:6819","nonce":14660}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":14660},{"type":"v1","addr":"172.21.15.61:6823","nonce":14660}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":14660},{"type":"v1","addr":"172.21.15.61:6821","nonce":14660}]},"public_addr":"172.21.15.61:6817/14660","cluster_addr":"172.21.15.61:6819/14660","heartbeat_back_addr":"172.21.15.61:6823/14660","heartbeat_front_addr":"172.21.15.61:6821/14660","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T06:10:57.694212+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T06:10:57.074765+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T06:10:57.729633+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-02-11T06:11:10.547 INFO:tasks.ceph.ceph_manager.c1:all up! 2021-02-11T06:11:10.547 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2021-02-11T06:11:10.850 INFO:teuthology.orchestra.run.smithi061.stdout: 2021-02-11T06:11:10.851 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":16,"fsid":"043de53e-a645-454b-bce7-c489302846ce","created":"2021-02-11T06:10:51.924984+0000","modified":"2021-02-11T06:11:09.909847+0000","last_up_change":"2021-02-11T06:10:59.194103+0000","last_in_change":"2021-02-11T06:10:59.194103+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-02-11T06:10:59.208467+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-02-11T06:11:07.282762+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"0d97d5d6-5f78-4266-9bb1-8f868bd62e54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":14659},{"type":"v1","addr":"172.21.15.61:6809","nonce":14659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":14659},{"type":"v1","addr":"172.21.15.61:6811","nonce":14659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":14659},{"type":"v1","addr":"172.21.15.61:6815","nonce":14659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":14659},{"type":"v1","addr":"172.21.15.61:6813","nonce":14659}]},"public_addr":"172.21.15.61:6809/14659","cluster_addr":"172.21.15.61:6811/14659","heartbeat_back_addr":"172.21.15.61:6815/14659","heartbeat_front_addr":"172.21.15.61:6813/14659","state":["exists","up"]},{"osd":1,"uuid":"2b921cd0-ae04-4e00-abee-e6327aaff01f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":14661},{"type":"v1","addr":"172.21.15.61:6801","nonce":14661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":14661},{"type":"v1","addr":"172.21.15.61:6803","nonce":14661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":14661},{"type":"v1","addr":"172.21.15.61:6807","nonce":14661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":14661},{"type":"v1","addr":"172.21.15.61:6805","nonce":14661}]},"public_addr":"172.21.15.61:6801/14661","cluster_addr":"172.21.15.61:6803/14661","heartbeat_back_addr":"172.21.15.61:6807/14661","heartbeat_front_addr":"172.21.15.61:6805/14661","state":["exists","up"]},{"osd":2,"uuid":"1d1b38e0-b970-4074-9525-26fef2f47dad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":14660},{"type":"v1","addr":"172.21.15.61:6817","nonce":14660}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":14660},{"type":"v1","addr":"172.21.15.61:6819","nonce":14660}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":14660},{"type":"v1","addr":"172.21.15.61:6823","nonce":14660}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":14660},{"type":"v1","addr":"172.21.15.61:6821","nonce":14660}]},"public_addr":"172.21.15.61:6817/14660","cluster_addr":"172.21.15.61:6819/14660","heartbeat_back_addr":"172.21.15.61:6823/14660","heartbeat_front_addr":"172.21.15.61:6821/14660","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T06:10:57.694212+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T06:10:57.074765+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T06:10:57.729633+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-02-11T06:11:10.865 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.0 flush_pg_stats 2021-02-11T06:11:11.035 INFO:teuthology.orchestra.run.smithi061.stdout:38654705668 2021-02-11T06:11:11.036 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.1 flush_pg_stats 2021-02-11T06:11:11.209 INFO:teuthology.orchestra.run.smithi061.stdout:34359738373 2021-02-11T06:11:11.210 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.2 flush_pg_stats 2021-02-11T06:11:11.382 INFO:teuthology.orchestra.run.smithi061.stdout:38654705668 2021-02-11T06:11:11.383 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2021-02-11T06:11:11.692 INFO:teuthology.orchestra.run.smithi061.stdout:38654705666 2021-02-11T06:11:11.706 INFO:tasks.ceph.ceph_manager.c1:need seq 38654705668 got 38654705666 for osd.0 2021-02-11T06:11:12.707 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2021-02-11T06:11:13.022 INFO:teuthology.orchestra.run.smithi061.stdout:38654705668 2021-02-11T06:11:13.037 INFO:tasks.ceph.ceph_manager.c1:need seq 38654705668 got 38654705668 for osd.0 2021-02-11T06:11:13.037 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2021-02-11T06:11:13.338 INFO:teuthology.orchestra.run.smithi061.stdout:34359738373 2021-02-11T06:11:13.818 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738373 got 34359738373 for osd.1 2021-02-11T06:11:13.818 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2021-02-11T06:11:14.125 INFO:teuthology.orchestra.run.smithi061.stdout:38654705668 2021-02-11T06:11:14.140 INFO:tasks.ceph.ceph_manager.c1:need seq 38654705668 got 38654705668 for osd.2 2021-02-11T06:11:14.141 INFO:tasks.ceph.ceph_manager.c1:waiting for clean 2021-02-11T06:11:14.141 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2021-02-11T06:11:14.445 INFO:teuthology.orchestra.run.smithi061.stdout: 2021-02-11T06:11:14.447 INFO:teuthology.orchestra.run.smithi061.stderr:dumped all 2021-02-11T06:11:14.459 INFO:teuthology.orchestra.run.smithi061.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2021-02-11T06:11:12.917599+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":14816,"kb_used_data":392,"kb_used_omap":0,"kb_used_meta":14400,"kb_avail":314557984,"statfs":{"total":322122547200,"available":322107375616,"internally_reserved":0,"allocated":401408,"data_stored":113985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":14745600},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"8","reported_epoch":"15","state":"active+clean","last_fresh":"2021-02-11T06:11:08.917972+0000","last_change":"2021-02-11T06:11:08.917972+0000","last_active":"2021-02-11T06:11:08.917972+0000","last_peered":"2021-02-11T06:11:08.917972+0000","last_clean":"2021-02-11T06:11:08.917972+0000","last_became_active":"2021-02-11T06:11:08.916530+0000","last_became_peered":"2021-02-11T06:11:08.916530+0000","last_unstale":"2021-02-11T06:11:08.917972+0000","last_undegraded":"2021-02-11T06:11:08.917972+0000","last_fullsized":"2021-02-11T06:11:08.917972+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T06:11:07.896086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T06:11:07.896086+0000","last_clean_scrub_stamp":"2021-02-11T06:11:07.896086+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"8","reported_epoch":"15","state":"active+clean","last_fresh":"2021-02-11T06:11:08.915993+0000","last_change":"2021-02-11T06:11:08.915993+0000","last_active":"2021-02-11T06:11:08.915993+0000","last_peered":"2021-02-11T06:11:08.915993+0000","last_clean":"2021-02-11T06:11:08.915993+0000","last_became_active":"2021-02-11T06:11:08.915173+0000","last_became_peered":"2021-02-11T06:11:08.915173+0000","last_unstale":"2021-02-11T06:11:08.915993+0000","last_undegraded":"2021-02-11T06:11:08.915993+0000","last_fullsized":"2021-02-11T06:11:08.915993+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T06:11:07.896086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T06:11:07.896086+0000","last_clean_scrub_stamp":"2021-02-11T06:11:07.896086+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"8","reported_epoch":"15","state":"active+clean","last_fresh":"2021-02-11T06:11:08.917225+0000","last_change":"2021-02-11T06:11:08.917225+0000","last_active":"2021-02-11T06:11:08.917225+0000","last_peered":"2021-02-11T06:11:08.917225+0000","last_clean":"2021-02-11T06:11:08.917225+0000","last_became_active":"2021-02-11T06:11:08.916462+0000","last_became_peered":"2021-02-11T06:11:08.916462+0000","last_unstale":"2021-02-11T06:11:08.917225+0000","last_undegraded":"2021-02-11T06:11:08.917225+0000","last_fullsized":"2021-02-11T06:11:08.917225+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T06:11:07.896086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T06:11:07.896086+0000","last_clean_scrub_stamp":"2021-02-11T06:11:07.896086+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"8","reported_epoch":"15","state":"active+clean","last_fresh":"2021-02-11T06:11:08.920117+0000","last_change":"2021-02-11T06:11:08.920117+0000","last_active":"2021-02-11T06:11:08.920117+0000","last_peered":"2021-02-11T06:11:08.920117+0000","last_clean":"2021-02-11T06:11:08.920117+0000","last_became_active":"2021-02-11T06:11:08.919716+0000","last_became_peered":"2021-02-11T06:11:08.919716+0000","last_unstale":"2021-02-11T06:11:08.920117+0000","last_undegraded":"2021-02-11T06:11:08.920117+0000","last_fullsized":"2021-02-11T06:11:08.920117+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T06:11:07.896086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T06:11:07.896086+0000","last_clean_scrub_stamp":"2021-02-11T06:11:07.896086+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"12","reported_epoch":"14","state":"active+clean","last_fresh":"2021-02-11T06:11:07.904614+0000","last_change":"2021-02-11T06:11:02.560231+0000","last_active":"2021-02-11T06:11:07.904614+0000","last_peered":"2021-02-11T06:11:07.904614+0000","last_clean":"2021-02-11T06:11:07.904614+0000","last_became_active":"2021-02-11T06:11:02.559464+0000","last_became_peered":"2021-02-11T06:11:02.559464+0000","last_unstale":"2021-02-11T06:11:07.904614+0000","last_undegraded":"2021-02-11T06:11:07.904614+0000","last_fullsized":"2021-02-11T06:11:07.904614+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T06:11:00.197735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T06:11:00.197735+0000","last_clean_scrub_stamp":"2021-02-11T06:11:00.197735+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"8","reported_epoch":"15","state":"active+clean","last_fresh":"2021-02-11T06:11:08.918297+0000","last_change":"2021-02-11T06:11:08.918297+0000","last_active":"2021-02-11T06:11:08.918297+0000","last_peered":"2021-02-11T06:11:08.918297+0000","last_clean":"2021-02-11T06:11:08.918297+0000","last_became_active":"2021-02-11T06:11:08.917996+0000","last_became_peered":"2021-02-11T06:11:08.917996+0000","last_unstale":"2021-02-11T06:11:08.918297+0000","last_undegraded":"2021-02-11T06:11:08.918297+0000","last_fullsized":"2021-02-11T06:11:08.918297+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T06:11:07.896086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T06:11:07.896086+0000","last_clean_scrub_stamp":"2021-02-11T06:11:07.896086+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"8","reported_epoch":"15","state":"active+clean","last_fresh":"2021-02-11T06:11:08.919389+0000","last_change":"2021-02-11T06:11:08.919389+0000","last_active":"2021-02-11T06:11:08.919389+0000","last_peered":"2021-02-11T06:11:08.919389+0000","last_clean":"2021-02-11T06:11:08.919389+0000","last_became_active":"2021-02-11T06:11:08.919064+0000","last_became_peered":"2021-02-11T06:11:08.919064+0000","last_unstale":"2021-02-11T06:11:08.919389+0000","last_undegraded":"2021-02-11T06:11:08.919389+0000","last_fullsized":"2021-02-11T06:11:08.919389+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T06:11:07.896086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T06:11:07.896086+0000","last_clean_scrub_stamp":"2021-02-11T06:11:07.896086+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"9","reported_epoch":"15","state":"active+clean","last_fresh":"2021-02-11T06:11:08.921815+0000","last_change":"2021-02-11T06:11:08.918313+0000","last_active":"2021-02-11T06:11:08.921815+0000","last_peered":"2021-02-11T06:11:08.921815+0000","last_clean":"2021-02-11T06:11:08.921815+0000","last_became_active":"2021-02-11T06:11:08.917967+0000","last_became_peered":"2021-02-11T06:11:08.917967+0000","last_unstale":"2021-02-11T06:11:08.921815+0000","last_undegraded":"2021-02-11T06:11:08.921815+0000","last_fullsized":"2021-02-11T06:11:08.921815+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T06:11:07.896086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T06:11:07.896086+0000","last_clean_scrub_stamp":"2021-02-11T06:11:07.896086+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"8","reported_epoch":"15","state":"active+clean","last_fresh":"2021-02-11T06:11:08.916549+0000","last_change":"2021-02-11T06:11:08.916549+0000","last_active":"2021-02-11T06:11:08.916549+0000","last_peered":"2021-02-11T06:11:08.916549+0000","last_clean":"2021-02-11T06:11:08.916549+0000","last_became_active":"2021-02-11T06:11:08.915563+0000","last_became_peered":"2021-02-11T06:11:08.915563+0000","last_unstale":"2021-02-11T06:11:08.916549+0000","last_undegraded":"2021-02-11T06:11:08.916549+0000","last_fullsized":"2021-02-11T06:11:08.916549+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T06:11:07.896086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T06:11:07.896086+0000","last_clean_scrub_stamp":"2021-02-11T06:11:07.896086+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4936,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852664,"statfs":{"total":107374182400,"available":107369127936,"internally_reserved":0,"allocated":131072,"data_stored":36824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1639999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.603}]}]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4936,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852664,"statfs":{"total":107374182400,"available":107369127936,"internally_reserved":0,"allocated":131072,"data_stored":36824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.038}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6659999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705668,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4944,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852656,"statfs":{"total":107374182400,"available":107369119744,"internally_reserved":0,"allocated":139264,"data_stored":40337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1859999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.585}]}]}]}} 2021-02-11T06:11:14.460 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2021-02-11T06:11:14.793 INFO:teuthology.orchestra.run.smithi061.stdout: 2021-02-11T06:11:15.119 INFO:teuthology.orchestra.run.smithi061.stderr:dumped all 2021-02-11T06:11:15.119 INFO:teuthology.orchestra.run.smithi061.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2021-02-11T06:11:12.917599+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":14816,"kb_used_data":392,"kb_used_omap":0,"kb_used_meta":14400,"kb_avail":314557984,"statfs":{"total":322122547200,"available":322107375616,"internally_reserved":0,"allocated":401408,"data_stored":113985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":14745600},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"8","reported_epoch":"15","state":"active+clean","last_fresh":"2021-02-11T06:11:08.917972+0000","last_change":"2021-02-11T06:11:08.917972+0000","last_active":"2021-02-11T06:11:08.917972+0000","last_peered":"2021-02-11T06:11:08.917972+0000","last_clean":"2021-02-11T06:11:08.917972+0000","last_became_active":"2021-02-11T06:11:08.916530+0000","last_became_peered":"2021-02-11T06:11:08.916530+0000","last_unstale":"2021-02-11T06:11:08.917972+0000","last_undegraded":"2021-02-11T06:11:08.917972+0000","last_fullsized":"2021-02-11T06:11:08.917972+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T06:11:07.896086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T06:11:07.896086+0000","last_clean_scrub_stamp":"2021-02-11T06:11:07.896086+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"8","reported_epoch":"15","state":"active+clean","last_fresh":"2021-02-11T06:11:08.915993+0000","last_change":"2021-02-11T06:11:08.915993+0000","last_active":"2021-02-11T06:11:08.915993+0000","last_peered":"2021-02-11T06:11:08.915993+0000","last_clean":"2021-02-11T06:11:08.915993+0000","last_became_active":"2021-02-11T06:11:08.915173+0000","last_became_peered":"2021-02-11T06:11:08.915173+0000","last_unstale":"2021-02-11T06:11:08.915993+0000","last_undegraded":"2021-02-11T06:11:08.915993+0000","last_fullsized":"2021-02-11T06:11:08.915993+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T06:11:07.896086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T06:11:07.896086+0000","last_clean_scrub_stamp":"2021-02-11T06:11:07.896086+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"8","reported_epoch":"15","state":"active+clean","last_fresh":"2021-02-11T06:11:08.917225+0000","last_change":"2021-02-11T06:11:08.917225+0000","last_active":"2021-02-11T06:11:08.917225+0000","last_peered":"2021-02-11T06:11:08.917225+0000","last_clean":"2021-02-11T06:11:08.917225+0000","last_became_active":"2021-02-11T06:11:08.916462+0000","last_became_peered":"2021-02-11T06:11:08.916462+0000","last_unstale":"2021-02-11T06:11:08.917225+0000","last_undegraded":"2021-02-11T06:11:08.917225+0000","last_fullsized":"2021-02-11T06:11:08.917225+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T06:11:07.896086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T06:11:07.896086+0000","last_clean_scrub_stamp":"2021-02-11T06:11:07.896086+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"8","reported_epoch":"15","state":"active+clean","last_fresh":"2021-02-11T06:11:08.920117+0000","last_change":"2021-02-11T06:11:08.920117+0000","last_active":"2021-02-11T06:11:08.920117+0000","last_peered":"2021-02-11T06:11:08.920117+0000","last_clean":"2021-02-11T06:11:08.920117+0000","last_became_active":"2021-02-11T06:11:08.919716+0000","last_became_peered":"2021-02-11T06:11:08.919716+0000","last_unstale":"2021-02-11T06:11:08.920117+0000","last_undegraded":"2021-02-11T06:11:08.920117+0000","last_fullsized":"2021-02-11T06:11:08.920117+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T06:11:07.896086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T06:11:07.896086+0000","last_clean_scrub_stamp":"2021-02-11T06:11:07.896086+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"12","reported_epoch":"14","state":"active+clean","last_fresh":"2021-02-11T06:11:07.904614+0000","last_change":"2021-02-11T06:11:02.560231+0000","last_active":"2021-02-11T06:11:07.904614+0000","last_peered":"2021-02-11T06:11:07.904614+0000","last_clean":"2021-02-11T06:11:07.904614+0000","last_became_active":"2021-02-11T06:11:02.559464+0000","last_became_peered":"2021-02-11T06:11:02.559464+0000","last_unstale":"2021-02-11T06:11:07.904614+0000","last_undegraded":"2021-02-11T06:11:07.904614+0000","last_fullsized":"2021-02-11T06:11:07.904614+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T06:11:00.197735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T06:11:00.197735+0000","last_clean_scrub_stamp":"2021-02-11T06:11:00.197735+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"8","reported_epoch":"15","state":"active+clean","last_fresh":"2021-02-11T06:11:08.918297+0000","last_change":"2021-02-11T06:11:08.918297+0000","last_active":"2021-02-11T06:11:08.918297+0000","last_peered":"2021-02-11T06:11:08.918297+0000","last_clean":"2021-02-11T06:11:08.918297+0000","last_became_active":"2021-02-11T06:11:08.917996+0000","last_became_peered":"2021-02-11T06:11:08.917996+0000","last_unstale":"2021-02-11T06:11:08.918297+0000","last_undegraded":"2021-02-11T06:11:08.918297+0000","last_fullsized":"2021-02-11T06:11:08.918297+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T06:11:07.896086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T06:11:07.896086+0000","last_clean_scrub_stamp":"2021-02-11T06:11:07.896086+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"8","reported_epoch":"15","state":"active+clean","last_fresh":"2021-02-11T06:11:08.919389+0000","last_change":"2021-02-11T06:11:08.919389+0000","last_active":"2021-02-11T06:11:08.919389+0000","last_peered":"2021-02-11T06:11:08.919389+0000","last_clean":"2021-02-11T06:11:08.919389+0000","last_became_active":"2021-02-11T06:11:08.919064+0000","last_became_peered":"2021-02-11T06:11:08.919064+0000","last_unstale":"2021-02-11T06:11:08.919389+0000","last_undegraded":"2021-02-11T06:11:08.919389+0000","last_fullsized":"2021-02-11T06:11:08.919389+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T06:11:07.896086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T06:11:07.896086+0000","last_clean_scrub_stamp":"2021-02-11T06:11:07.896086+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"9","reported_epoch":"15","state":"active+clean","last_fresh":"2021-02-11T06:11:08.921815+0000","last_change":"2021-02-11T06:11:08.918313+0000","last_active":"2021-02-11T06:11:08.921815+0000","last_peered":"2021-02-11T06:11:08.921815+0000","last_clean":"2021-02-11T06:11:08.921815+0000","last_became_active":"2021-02-11T06:11:08.917967+0000","last_became_peered":"2021-02-11T06:11:08.917967+0000","last_unstale":"2021-02-11T06:11:08.921815+0000","last_undegraded":"2021-02-11T06:11:08.921815+0000","last_fullsized":"2021-02-11T06:11:08.921815+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T06:11:07.896086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T06:11:07.896086+0000","last_clean_scrub_stamp":"2021-02-11T06:11:07.896086+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"8","reported_epoch":"15","state":"active+clean","last_fresh":"2021-02-11T06:11:08.916549+0000","last_change":"2021-02-11T06:11:08.916549+0000","last_active":"2021-02-11T06:11:08.916549+0000","last_peered":"2021-02-11T06:11:08.916549+0000","last_clean":"2021-02-11T06:11:08.916549+0000","last_became_active":"2021-02-11T06:11:08.915563+0000","last_became_peered":"2021-02-11T06:11:08.915563+0000","last_unstale":"2021-02-11T06:11:08.916549+0000","last_undegraded":"2021-02-11T06:11:08.916549+0000","last_fullsized":"2021-02-11T06:11:08.916549+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T06:11:07.896086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T06:11:07.896086+0000","last_clean_scrub_stamp":"2021-02-11T06:11:07.896086+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4936,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852664,"statfs":{"total":107374182400,"available":107369127936,"internally_reserved":0,"allocated":131072,"data_stored":36824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1639999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.603}]}]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4936,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852664,"statfs":{"total":107374182400,"available":107369127936,"internally_reserved":0,"allocated":131072,"data_stored":36824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.038}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6659999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705668,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4944,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852656,"statfs":{"total":107374182400,"available":107369119744,"internally_reserved":0,"allocated":139264,"data_stored":40337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1859999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.585}]}]}]}} 2021-02-11T06:11:15.120 INFO:tasks.ceph.ceph_manager.c1:clean! 2021-02-11T06:11:15.120 INFO:tasks.ceph:Waiting until ceph cluster c1 is healthy... 2021-02-11T06:11:15.120 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy 2021-02-11T06:11:15.120 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 health --format=json 2021-02-11T06:11:15.478 INFO:teuthology.orchestra.run.smithi061.stdout: 2021-02-11T06:11:15.494 INFO:teuthology.orchestra.run.smithi061.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2021-02-11T06:11:15.494 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy done 2021-02-11T06:11:15.495 INFO:teuthology.run_tasks:Running task ceph... 2021-02-11T06:11:15.501 INFO:tasks.ceph:Creating ceph cluster c2... 2021-02-11T06:11:15.501 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'cpu_profile': set(), 'cluster': 'c2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2021-02-11T06:11:15.502 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/aemerson-2021-02-11_05:45:06-rgw-wip-easter-bunny-distro-basic-smithi/5874443', 'branch': 'wip-easter-bunny', 'description': 'rgw/multisite/{clusters frontend/beast omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind}', 'email': 'aemerson@redhat.com', 'first_in_suite': False, 'job_id': '5874443', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'aemerson-2021-02-11_05:45:06-rgw-wip-easter-bunny-distro-basic-smithi', 'nuke-on-error': True, 'os_type': 'ubuntu', 'overrides': {'admin_socket': {'branch': 'wip-easter-bunny'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '84a8025e5a892a6879f75e17e03444bf443eae4b', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': '84a8025e5a892a6879f75e17e03444bf443eae4b'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}, {'endpoints': ['c2.client.1'], 'is_pubsub': True, 'name': 'test-zone4'}]}]}, 'workunit': {'branch': 'wip-easter-bunny', 'sha1': '84a8025e5a892a6879f75e17e03444bf443eae4b'}}, 'owner': 'scheduled_aemerson@teuthology', 'priority': 70, 'repo': 'git://git.ceph.com/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'sha1': '84a8025e5a892a6879f75e17e03444bf443eae4b', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'wip-easter-bunny', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_84a8025e5a892a6879f75e17e03444bf443eae4b/qa', 'suite_relpath': 'qa', 'suite_repo': 'git://git.ceph.com/ceph-ci.git', 'suite_sha1': '84a8025e5a892a6879f75e17e03444bf443eae4b', 'targets': {'smithi061.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9nFZMnqNQMV9nVvBUk/hdcUhFPn2XCz9ZkCiQbtge7HeikSAfUCyOCHSfz0S1pmOcIbc6LkEm4xWkAeLB9/pdSZ+PzVLBUanYR6RkuRKO4xTYn+JavSZmJCi3NQ5NcsBR04RRvp/saikmxQmmdlwsgajmkmavyzkL90CEquTby7jL4PERDRhH/qpnwlzPWgaaiea3k9Yh41Jvtd1xQi0+1IsNGsmoVgvfz90gsgnXHmuuYXUTC9xbXtlZnq1pOmBAtWlaf0n/YpZUMki+jsM6TOwG3OOGZ2m69P7yNNbjhZiBmDSi0lBcx7cMG7lyYWJrFO+6a+MvQA1L/Z1hR14X', 'smithi172.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKc1kIvUvofa8RKQkSgMIYhvCE2w7RZG4ZgcVx5kPL/pTrHdUfID6cH6NblNsgR97jYHYb7olU29Qx8ZIgSSI3zbdAxGNt1IgvEfG58vL1U2g0Ln3iIV/ZRIzyl22IbRjb4UNstYPCxc5lX9Ni+ngwwE7XmvflhXJAL5Ykpq74qpcvhcG0T/JCADGX2q+LhAUSmgb6vYzWdwz1GhObmV2W80cMQBpmsoffW5w3w7uu6rHdNA0rK1C9x0/bqDMp1TUViv3pe7jxD3JlufcDDIszdtwc2s4n1w0HhzbQmV1tJZUD1tKjtTOoqWuFdMb9t//7ZKqjatFUjJzKc3ZWv86X'}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '84a8025e5a892a6879f75e17e03444bf443eae4b', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '84a8025e5a892a6879f75e17e03444bf443eae4b', 'wait-for-scrub': False}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology_branch': 'master', 'timestamp': '2021-02-11_05:45:06', 'tube': 'smithi', 'user': 'aemerson', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.29888'} 2021-02-11T06:11:15.502 DEBUG:teuthology.orchestra.run.smithi061:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2021-02-11T06:11:15.504 DEBUG:teuthology.orchestra.run.smithi172:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2021-02-11T06:11:15.511 DEBUG:teuthology.orchestra.run.smithi061:> sudo install -d -m0777 -- /var/run/ceph 2021-02-11T06:11:15.549 DEBUG:teuthology.orchestra.run.smithi172:> sudo install -d -m0777 -- /var/run/ceph 2021-02-11T06:11:15.569 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2021-02-11T06:11:15.569 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/scratch_devs of=/dev/stdout 2021-02-11T06:11:15.616 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-02-11T06:11:15.616 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-02-11T06:11:15.665 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2021-02-11T06:11:15.665 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-11T06:11:15.665 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 6h/6d Inode: 342 Links: 1 2021-02-11T06:11:15.665 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-11T06:11:15.665 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2021-02-11 06:07:00.068425185 +0000 2021-02-11T06:11:15.665 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2021-02-11 06:06:59.976425168 +0000 2021-02-11T06:11:15.665 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2021-02-11 06:06:59.976425168 +0000 2021-02-11T06:11:15.665 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: - 2021-02-11T06:11:15.674 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2021-02-11T06:11:15.675 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2021-02-11T06:11:15.675 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000406404 s, 1.3 MB/s 2021-02-11T06:11:15.678 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-02-11T06:11:15.729 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2021-02-11T06:11:15.729 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-11T06:11:15.729 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 6h/6d Inode: 348 Links: 1 2021-02-11T06:11:15.729 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-11T06:11:15.729 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2021-02-11 06:07:00.220425213 +0000 2021-02-11T06:11:15.729 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2021-02-11 06:07:00.216425213 +0000 2021-02-11T06:11:15.729 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2021-02-11 06:07:00.216425213 +0000 2021-02-11T06:11:15.730 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: - 2021-02-11T06:11:15.739 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2021-02-11T06:11:15.739 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2021-02-11T06:11:15.739 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000390563 s, 1.3 MB/s 2021-02-11T06:11:15.744 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-02-11T06:11:15.797 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2021-02-11T06:11:15.797 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-11T06:11:15.797 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 6h/6d Inode: 354 Links: 1 2021-02-11T06:11:15.797 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-11T06:11:15.797 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2021-02-11 06:07:00.580425280 +0000 2021-02-11T06:11:15.797 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2021-02-11 06:07:00.488425263 +0000 2021-02-11T06:11:15.797 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2021-02-11 06:07:00.488425263 +0000 2021-02-11T06:11:15.797 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: - 2021-02-11T06:11:15.806 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2021-02-11T06:11:15.807 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2021-02-11T06:11:15.807 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000359785 s, 1.4 MB/s 2021-02-11T06:11:15.811 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-02-11T06:11:15.865 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2021-02-11T06:11:15.865 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-11T06:11:15.865 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 6h/6d Inode: 360 Links: 1 2021-02-11T06:11:15.865 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-11T06:11:15.865 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2021-02-11 06:07:00.812425324 +0000 2021-02-11T06:11:15.865 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2021-02-11 06:07:00.708425304 +0000 2021-02-11T06:11:15.865 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2021-02-11 06:07:00.708425304 +0000 2021-02-11T06:11:15.865 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: - 2021-02-11T06:11:15.874 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2021-02-11T06:11:15.875 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2021-02-11T06:11:15.875 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.00036219 s, 1.4 MB/s 2021-02-11T06:11:15.879 INFO:tasks.ceph:osd dev map: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2021-02-11T06:11:15.879 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi172.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2021-02-11T06:11:15.879 INFO:tasks.ceph:Generating config... 2021-02-11T06:11:15.880 INFO:tasks.ceph:[client] debug rgw = 20 2021-02-11T06:11:15.881 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2021-02-11T06:11:15.881 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2021-02-11T06:11:15.881 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-02-11T06:11:15.881 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2021-02-11T06:11:15.881 INFO:tasks.ceph:[client] rgw data log num shards = 4 2021-02-11T06:11:15.881 INFO:tasks.ceph:[client] rgw md log max shards = 4 2021-02-11T06:11:15.881 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2021-02-11T06:11:15.881 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2021-02-11T06:11:15.881 INFO:tasks.ceph:[client] setgroup = ceph 2021-02-11T06:11:15.881 INFO:tasks.ceph:[client] setuser = ceph 2021-02-11T06:11:15.881 INFO:tasks.ceph:[global] osd heartbeat grace = 40 2021-02-11T06:11:15.881 INFO:tasks.ceph:[mgr] debug mgr = 20 2021-02-11T06:11:15.881 INFO:tasks.ceph:[mgr] debug ms = 1 2021-02-11T06:11:15.881 INFO:tasks.ceph:[mon] debug mon = 20 2021-02-11T06:11:15.882 INFO:tasks.ceph:[mon] debug ms = 1 2021-02-11T06:11:15.882 INFO:tasks.ceph:[mon] debug paxos = 20 2021-02-11T06:11:15.882 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2021-02-11T06:11:15.882 INFO:tasks.ceph:[osd] debug filestore = 20 2021-02-11T06:11:15.882 INFO:tasks.ceph:[osd] debug journal = 20 2021-02-11T06:11:15.882 INFO:tasks.ceph:[osd] debug ms = 20 2021-02-11T06:11:15.882 INFO:tasks.ceph:[osd] debug osd = 25 2021-02-11T06:11:15.882 INFO:tasks.ceph:[osd] osd fast shutdown = False 2021-02-11T06:11:15.882 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2021-02-11T06:11:15.882 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2021-02-11T06:11:15.882 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2021-02-11T06:11:15.882 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2021-02-11T06:11:15.883 INFO:tasks.ceph:Setting up c2.mon.a... 2021-02-11T06:11:15.883 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c2.keyring 2021-02-11T06:11:15.958 INFO:teuthology.orchestra.run.smithi172.stdout:creating /etc/ceph/c2.keyring 2021-02-11T06:11:15.960 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c2.keyring 2021-02-11T06:11:16.036 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 0644 /etc/ceph/c2.keyring 2021-02-11T06:11:16.093 DEBUG:tasks.ceph:Ceph mon addresses: [('c2.mon.a', '172.21.15.172')] 2021-02-11T06:11:16.093 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.172', 'osd heartbeat grace': 40}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'mon osd crush smoke test': False}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2021-02-11T06:11:16.094 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2021-02-11T06:11:16.094 DEBUG:teuthology.orchestra.run.smithi172:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2021-02-11T06:11:16.141 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.172 --print /home/ubuntu/cephtest/c2.monmap 2021-02-11T06:11:28.220 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:11:28.220 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:11:28.221 INFO:teuthology.orchestra.run.smithi172.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c2.monmap 2021-02-11T06:11:28.221 INFO:teuthology.orchestra.run.smithi172.stdout:monmaptool: generated fsid 0d6e87fb-daa0-4c85-a370-bf71998465da 2021-02-11T06:11:28.221 INFO:teuthology.orchestra.run.smithi172.stdout:epoch 0 2021-02-11T06:11:28.221 INFO:teuthology.orchestra.run.smithi172.stdout:fsid 0d6e87fb-daa0-4c85-a370-bf71998465da 2021-02-11T06:11:28.221 INFO:teuthology.orchestra.run.smithi172.stdout:last_changed 2021-02-11T06:11:28.226041+0000 2021-02-11T06:11:28.221 INFO:teuthology.orchestra.run.smithi172.stdout:created 2021-02-11T06:11:28.226041+0000 2021-02-11T06:11:28.221 INFO:teuthology.orchestra.run.smithi172.stdout:min_mon_release 0 (unknown) 2021-02-11T06:11:28.221 INFO:teuthology.orchestra.run.smithi172.stdout:election_strategy: 1 2021-02-11T06:11:28.221 INFO:teuthology.orchestra.run.smithi172.stdout:0: [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] mon.a 2021-02-11T06:11:28.222 INFO:teuthology.orchestra.run.smithi172.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c2.monmap (1 monitors) 2021-02-11T06:11:28.222 DEBUG:teuthology.orchestra.run.smithi172:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2021-02-11T06:11:28.272 INFO:tasks.ceph:Writing /etc/ceph/c2.conf for FSID 0d6e87fb-daa0-4c85-a370-bf71998465da... 2021-02-11T06:11:28.273 DEBUG:teuthology.orchestra.run.smithi061:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2021-02-11T06:11:28.275 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2021-02-11T06:11:28.317 INFO:teuthology.orchestra.run.smithi061.stdout:[global] 2021-02-11T06:11:28.317 INFO:teuthology.orchestra.run.smithi061.stdout: chdir = "" 2021-02-11T06:11:28.317 INFO:teuthology.orchestra.run.smithi061.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-02-11T06:11:28.317 INFO:teuthology.orchestra.run.smithi061.stdout: auth supported = cephx 2021-02-11T06:11:28.317 INFO:teuthology.orchestra.run.smithi061.stdout: 2021-02-11T06:11:28.317 INFO:teuthology.orchestra.run.smithi061.stdout: filestore xattr use omap = true 2021-02-11T06:11:28.317 INFO:teuthology.orchestra.run.smithi061.stdout: 2021-02-11T06:11:28.317 INFO:teuthology.orchestra.run.smithi061.stdout: mon clock drift allowed = 1.000 2021-02-11T06:11:28.317 INFO:teuthology.orchestra.run.smithi061.stdout: 2021-02-11T06:11:28.318 INFO:teuthology.orchestra.run.smithi061.stdout: osd crush chooseleaf type = 0 2021-02-11T06:11:28.318 INFO:teuthology.orchestra.run.smithi061.stdout: auth debug = true 2021-02-11T06:11:28.318 INFO:teuthology.orchestra.run.smithi061.stdout: 2021-02-11T06:11:28.318 INFO:teuthology.orchestra.run.smithi061.stdout: ms die on old message = true 2021-02-11T06:11:28.318 INFO:teuthology.orchestra.run.smithi061.stdout: ms die on bug = true 2021-02-11T06:11:28.318 INFO:teuthology.orchestra.run.smithi061.stdout: 2021-02-11T06:11:28.318 INFO:teuthology.orchestra.run.smithi061.stdout: mon max pg per osd = 10000 # >= luminous 2021-02-11T06:11:28.318 INFO:teuthology.orchestra.run.smithi061.stdout: mon pg warn max object skew = 0 2021-02-11T06:11:28.318 INFO:teuthology.orchestra.run.smithi061.stdout: 2021-02-11T06:11:28.318 INFO:teuthology.orchestra.run.smithi061.stdout: # disable pg_autoscaler by default for new pools 2021-02-11T06:11:28.318 INFO:teuthology.orchestra.run.smithi061.stdout: osd_pool_default_pg_autoscale_mode = off 2021-02-11T06:11:28.318 INFO:teuthology.orchestra.run.smithi061.stdout: 2021-02-11T06:11:28.318 INFO:teuthology.orchestra.run.smithi061.stdout: osd pool default size = 2 2021-02-11T06:11:28.318 INFO:teuthology.orchestra.run.smithi061.stdout: 2021-02-11T06:11:28.319 INFO:teuthology.orchestra.run.smithi061.stdout: mon osd allow primary affinity = true 2021-02-11T06:11:28.319 INFO:teuthology.orchestra.run.smithi061.stdout: mon osd allow pg remap = true 2021-02-11T06:11:28.319 INFO:teuthology.orchestra.run.smithi061.stdout: mon warn on legacy crush tunables = false 2021-02-11T06:11:28.319 INFO:teuthology.orchestra.run.smithi061.stdout: mon warn on crush straw calc version zero = false 2021-02-11T06:11:28.319 INFO:teuthology.orchestra.run.smithi061.stdout: mon warn on no sortbitwise = false 2021-02-11T06:11:28.319 INFO:teuthology.orchestra.run.smithi061.stdout: mon warn on osd down out interval zero = false 2021-02-11T06:11:28.319 INFO:teuthology.orchestra.run.smithi061.stdout: mon warn on too few osds = false 2021-02-11T06:11:28.319 INFO:teuthology.orchestra.run.smithi061.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-02-11T06:11:28.319 INFO:teuthology.orchestra.run.smithi061.stdout: mon_warn_on_pool_no_redundancy = false 2021-02-11T06:11:28.319 INFO:teuthology.orchestra.run.smithi061.stdout: mon_allow_pool_size_one = true 2021-02-11T06:11:28.319 INFO:teuthology.orchestra.run.smithi061.stdout: 2021-02-11T06:11:28.319 INFO:teuthology.orchestra.run.smithi061.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-02-11T06:11:28.319 INFO:teuthology.orchestra.run.smithi061.stdout: 2021-02-11T06:11:28.319 INFO:teuthology.orchestra.run.smithi061.stdout: osd default data pool replay window = 5 2021-02-11T06:11:28.319 INFO:teuthology.orchestra.run.smithi061.stdout: 2021-02-11T06:11:28.320 INFO:teuthology.orchestra.run.smithi061.stdout: mon allow pool delete = true 2021-02-11T06:11:28.320 INFO:teuthology.orchestra.run.smithi061.stdout: 2021-02-11T06:11:28.320 INFO:teuthology.orchestra.run.smithi061.stdout: mon cluster log file level = debug 2021-02-11T06:11:28.320 INFO:teuthology.orchestra.run.smithi061.stdout: debug asserts on shutdown = true 2021-02-11T06:11:28.320 INFO:teuthology.orchestra.run.smithi061.stdout: mon health detail to clog = false 2021-02-11T06:11:28.320 INFO:teuthology.orchestra.run.smithi061.stdout: mon host = 172.21.15.172 2021-02-11T06:11:28.320 INFO:teuthology.orchestra.run.smithi061.stdout: osd heartbeat grace = 40 2021-02-11T06:11:28.320 INFO:teuthology.orchestra.run.smithi061.stdout: fsid = 0d6e87fb-daa0-4c85-a370-bf71998465da 2021-02-11T06:11:28.320 INFO:teuthology.orchestra.run.smithi061.stdout: 2021-02-11T06:11:28.320 INFO:teuthology.orchestra.run.smithi061.stdout:[osd] 2021-02-11T06:11:28.320 INFO:teuthology.orchestra.run.smithi061.stdout: osd journal size = 100 2021-02-11T06:11:28.320 INFO:teuthology.orchestra.run.smithi061.stdout: 2021-02-11T06:11:28.320 INFO:teuthology.orchestra.run.smithi061.stdout: osd scrub load threshold = 5.0 2021-02-11T06:11:28.320 INFO:teuthology.orchestra.run.smithi061.stdout: osd scrub max interval = 600 2021-02-11T06:11:28.321 INFO:teuthology.orchestra.run.smithi061.stdout: 2021-02-11T06:11:28.321 INFO:teuthology.orchestra.run.smithi061.stdout: osd recover clone overlap = true 2021-02-11T06:11:28.321 INFO:teuthology.orchestra.run.smithi061.stdout: osd recovery max chunk = 1048576 2021-02-11T06:11:28.321 INFO:teuthology.orchestra.run.smithi061.stdout: 2021-02-11T06:11:28.321 INFO:teuthology.orchestra.run.smithi061.stdout: osd debug shutdown = true 2021-02-11T06:11:28.321 INFO:teuthology.orchestra.run.smithi061.stdout: osd debug op order = true 2021-02-11T06:11:28.321 INFO:teuthology.orchestra.run.smithi061.stdout: osd debug verify stray on activate = true 2021-02-11T06:11:28.321 INFO:teuthology.orchestra.run.smithi061.stdout: 2021-02-11T06:11:28.321 INFO:teuthology.orchestra.run.smithi061.stdout: osd open classes on start = true 2021-02-11T06:11:28.321 INFO:teuthology.orchestra.run.smithi061.stdout: osd debug pg log writeout = true 2021-02-11T06:11:28.321 INFO:teuthology.orchestra.run.smithi061.stdout: 2021-02-11T06:11:28.321 INFO:teuthology.orchestra.run.smithi061.stdout: osd deep scrub update digest min age = 30 2021-02-11T06:11:28.321 INFO:teuthology.orchestra.run.smithi061.stdout: 2021-02-11T06:11:28.321 INFO:teuthology.orchestra.run.smithi061.stdout: osd map max advance = 10 2021-02-11T06:11:28.322 INFO:teuthology.orchestra.run.smithi061.stdout: 2021-02-11T06:11:28.322 INFO:teuthology.orchestra.run.smithi061.stdout: journal zero on create = true 2021-02-11T06:11:28.322 INFO:teuthology.orchestra.run.smithi061.stdout: 2021-02-11T06:11:28.322 INFO:teuthology.orchestra.run.smithi061.stdout: filestore ondisk finisher threads = 3 2021-02-11T06:11:28.322 INFO:teuthology.orchestra.run.smithi061.stdout: filestore apply finisher threads = 3 2021-02-11T06:11:28.322 INFO:teuthology.orchestra.run.smithi061.stdout: 2021-02-11T06:11:28.322 INFO:teuthology.orchestra.run.smithi061.stdout: bdev debug aio = true 2021-02-11T06:11:28.322 INFO:teuthology.orchestra.run.smithi061.stdout: osd debug misdirected ops = true 2021-02-11T06:11:28.322 INFO:teuthology.orchestra.run.smithi061.stdout: debug filestore = 20 2021-02-11T06:11:28.322 INFO:teuthology.orchestra.run.smithi061.stdout: debug journal = 20 2021-02-11T06:11:28.322 INFO:teuthology.orchestra.run.smithi061.stdout: debug ms = 20 2021-02-11T06:11:28.322 INFO:teuthology.orchestra.run.smithi061.stdout: debug osd = 25 2021-02-11T06:11:28.322 INFO:teuthology.orchestra.run.smithi061.stdout: osd fast shutdown = False 2021-02-11T06:11:28.322 INFO:teuthology.orchestra.run.smithi061.stdout: rocksdb delete range threshold = 0 2021-02-11T06:11:28.323 INFO:teuthology.orchestra.run.smithi061.stdout: 2021-02-11T06:11:28.323 INFO:teuthology.orchestra.run.smithi061.stdout:[mgr] 2021-02-11T06:11:28.323 INFO:teuthology.orchestra.run.smithi061.stdout: debug ms = 1 2021-02-11T06:11:28.323 INFO:teuthology.orchestra.run.smithi061.stdout: debug mgr = 20 2021-02-11T06:11:28.323 INFO:teuthology.orchestra.run.smithi061.stdout: debug mon = 20 2021-02-11T06:11:28.323 INFO:teuthology.orchestra.run.smithi061.stdout: debug auth = 20 2021-02-11T06:11:28.323 INFO:teuthology.orchestra.run.smithi061.stdout: mon reweight min pgs per osd = 4 2021-02-11T06:11:28.323 INFO:teuthology.orchestra.run.smithi061.stdout: mon reweight min bytes per osd = 10 2021-02-11T06:11:28.323 INFO:teuthology.orchestra.run.smithi061.stdout: mgr/telemetry/nag = false 2021-02-11T06:11:28.323 INFO:teuthology.orchestra.run.smithi061.stdout: 2021-02-11T06:11:28.323 INFO:teuthology.orchestra.run.smithi061.stdout:[mon] 2021-02-11T06:11:28.323 INFO:teuthology.orchestra.run.smithi061.stdout: debug ms = 1 2021-02-11T06:11:28.323 INFO:teuthology.orchestra.run.smithi061.stdout: debug mon = 20 2021-02-11T06:11:28.323 INFO:teuthology.orchestra.run.smithi061.stdout: debug paxos = 20 2021-02-11T06:11:28.323 INFO:teuthology.orchestra.run.smithi061.stdout: debug auth = 20 2021-02-11T06:11:28.324 INFO:teuthology.orchestra.run.smithi061.stdout: mon data avail warn = 5 2021-02-11T06:11:28.324 INFO:teuthology.orchestra.run.smithi061.stdout: mon mgr mkfs grace = 240 2021-02-11T06:11:28.324 INFO:teuthology.orchestra.run.smithi061.stdout: mon reweight min pgs per osd = 4 2021-02-11T06:11:28.324 INFO:teuthology.orchestra.run.smithi061.stdout: mon osd reporter subtree level = osd 2021-02-11T06:11:28.324 INFO:teuthology.orchestra.run.smithi061.stdout: mon osd prime pg temp = true 2021-02-11T06:11:28.324 INFO:teuthology.orchestra.run.smithi061.stdout: mon reweight min bytes per osd = 10 2021-02-11T06:11:28.324 INFO:teuthology.orchestra.run.smithi061.stdout: mon osd crush smoke test = False 2021-02-11T06:11:28.324 INFO:teuthology.orchestra.run.smithi061.stdout: 2021-02-11T06:11:28.324 INFO:teuthology.orchestra.run.smithi061.stdout:[client] 2021-02-11T06:11:28.324 INFO:teuthology.orchestra.run.smithi061.stdout: rgw cache enabled = true 2021-02-11T06:11:28.324 INFO:teuthology.orchestra.run.smithi061.stdout: rgw enable ops log = true 2021-02-11T06:11:28.324 INFO:teuthology.orchestra.run.smithi061.stdout: rgw enable usage log = true 2021-02-11T06:11:28.324 INFO:teuthology.orchestra.run.smithi061.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-02-11T06:11:28.324 INFO:teuthology.orchestra.run.smithi061.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-02-11T06:11:28.325 INFO:teuthology.orchestra.run.smithi061.stdout: debug rgw = 20 2021-02-11T06:11:28.325 INFO:teuthology.orchestra.run.smithi061.stdout: rgw crypt require ssl = False 2021-02-11T06:11:28.325 INFO:teuthology.orchestra.run.smithi061.stdout: rgw crypt s3 kms backend = testing 2021-02-11T06:11:28.325 INFO:teuthology.orchestra.run.smithi061.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-02-11T06:11:28.325 INFO:teuthology.orchestra.run.smithi061.stdout: rgw curl low speed time = 300 2021-02-11T06:11:28.325 INFO:teuthology.orchestra.run.smithi061.stdout: rgw data log num shards = 4 2021-02-11T06:11:28.325 INFO:teuthology.orchestra.run.smithi061.stdout: rgw md log max shards = 4 2021-02-11T06:11:28.325 INFO:teuthology.orchestra.run.smithi061.stdout: rgw sync log trim interval = 0 2021-02-11T06:11:28.325 INFO:teuthology.orchestra.run.smithi061.stdout: rgw sync obj etag verify = True 2021-02-11T06:11:28.325 INFO:teuthology.orchestra.run.smithi061.stdout: setgroup = ceph 2021-02-11T06:11:28.325 INFO:teuthology.orchestra.run.smithi061.stdout: setuser = ceph 2021-02-11T06:11:28.325 INFO:teuthology.orchestra.run.smithi061.stdout:[mon.a] 2021-02-11T06:11:28.325 INFO:teuthology.orchestra.run.smithi061.stdout:[osd.0] 2021-02-11T06:11:28.325 INFO:teuthology.orchestra.run.smithi061.stdout: osd_max_omap_entries_per_request = 10 2021-02-11T06:11:28.325 INFO:teuthology.orchestra.run.smithi061.stdout:[osd.1] 2021-02-11T06:11:28.326 INFO:teuthology.orchestra.run.smithi061.stdout: osd_max_omap_entries_per_request = 1000 2021-02-11T06:11:28.326 INFO:teuthology.orchestra.run.smithi061.stdout:[osd.2] 2021-02-11T06:11:28.326 INFO:teuthology.orchestra.run.smithi061.stdout: osd_max_omap_entries_per_request = 10000 2021-02-11T06:11:28.346 INFO:teuthology.orchestra.run.smithi172.stdout:[global] 2021-02-11T06:11:28.346 INFO:teuthology.orchestra.run.smithi172.stdout: chdir = "" 2021-02-11T06:11:28.346 INFO:teuthology.orchestra.run.smithi172.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-02-11T06:11:28.346 INFO:teuthology.orchestra.run.smithi172.stdout: auth supported = cephx 2021-02-11T06:11:28.346 INFO:teuthology.orchestra.run.smithi172.stdout: 2021-02-11T06:11:28.346 INFO:teuthology.orchestra.run.smithi172.stdout: filestore xattr use omap = true 2021-02-11T06:11:28.346 INFO:teuthology.orchestra.run.smithi172.stdout: 2021-02-11T06:11:28.346 INFO:teuthology.orchestra.run.smithi172.stdout: mon clock drift allowed = 1.000 2021-02-11T06:11:28.346 INFO:teuthology.orchestra.run.smithi172.stdout: 2021-02-11T06:11:28.346 INFO:teuthology.orchestra.run.smithi172.stdout: osd crush chooseleaf type = 0 2021-02-11T06:11:28.347 INFO:teuthology.orchestra.run.smithi172.stdout: auth debug = true 2021-02-11T06:11:28.347 INFO:teuthology.orchestra.run.smithi172.stdout: 2021-02-11T06:11:28.347 INFO:teuthology.orchestra.run.smithi172.stdout: ms die on old message = true 2021-02-11T06:11:28.347 INFO:teuthology.orchestra.run.smithi172.stdout: ms die on bug = true 2021-02-11T06:11:28.347 INFO:teuthology.orchestra.run.smithi172.stdout: 2021-02-11T06:11:28.347 INFO:teuthology.orchestra.run.smithi172.stdout: mon max pg per osd = 10000 # >= luminous 2021-02-11T06:11:28.347 INFO:teuthology.orchestra.run.smithi172.stdout: mon pg warn max object skew = 0 2021-02-11T06:11:28.347 INFO:teuthology.orchestra.run.smithi172.stdout: 2021-02-11T06:11:28.347 INFO:teuthology.orchestra.run.smithi172.stdout: # disable pg_autoscaler by default for new pools 2021-02-11T06:11:28.347 INFO:teuthology.orchestra.run.smithi172.stdout: osd_pool_default_pg_autoscale_mode = off 2021-02-11T06:11:28.347 INFO:teuthology.orchestra.run.smithi172.stdout: 2021-02-11T06:11:28.348 INFO:teuthology.orchestra.run.smithi172.stdout: osd pool default size = 2 2021-02-11T06:11:28.348 INFO:teuthology.orchestra.run.smithi172.stdout: 2021-02-11T06:11:28.348 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd allow primary affinity = true 2021-02-11T06:11:28.348 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd allow pg remap = true 2021-02-11T06:11:28.348 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on legacy crush tunables = false 2021-02-11T06:11:28.348 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on crush straw calc version zero = false 2021-02-11T06:11:28.348 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on no sortbitwise = false 2021-02-11T06:11:28.348 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on osd down out interval zero = false 2021-02-11T06:11:28.348 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on too few osds = false 2021-02-11T06:11:28.348 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-02-11T06:11:28.348 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_pool_no_redundancy = false 2021-02-11T06:11:28.348 INFO:teuthology.orchestra.run.smithi172.stdout: mon_allow_pool_size_one = true 2021-02-11T06:11:28.348 INFO:teuthology.orchestra.run.smithi172.stdout: 2021-02-11T06:11:28.348 INFO:teuthology.orchestra.run.smithi172.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-02-11T06:11:28.349 INFO:teuthology.orchestra.run.smithi172.stdout: 2021-02-11T06:11:28.349 INFO:teuthology.orchestra.run.smithi172.stdout: osd default data pool replay window = 5 2021-02-11T06:11:28.349 INFO:teuthology.orchestra.run.smithi172.stdout: 2021-02-11T06:11:28.349 INFO:teuthology.orchestra.run.smithi172.stdout: mon allow pool delete = true 2021-02-11T06:11:28.349 INFO:teuthology.orchestra.run.smithi172.stdout: 2021-02-11T06:11:28.349 INFO:teuthology.orchestra.run.smithi172.stdout: mon cluster log file level = debug 2021-02-11T06:11:28.349 INFO:teuthology.orchestra.run.smithi172.stdout: debug asserts on shutdown = true 2021-02-11T06:11:28.349 INFO:teuthology.orchestra.run.smithi172.stdout: mon health detail to clog = false 2021-02-11T06:11:28.349 INFO:teuthology.orchestra.run.smithi172.stdout: mon host = 172.21.15.172 2021-02-11T06:11:28.349 INFO:teuthology.orchestra.run.smithi172.stdout: osd heartbeat grace = 40 2021-02-11T06:11:28.349 INFO:teuthology.orchestra.run.smithi172.stdout: fsid = 0d6e87fb-daa0-4c85-a370-bf71998465da 2021-02-11T06:11:28.349 INFO:teuthology.orchestra.run.smithi172.stdout: 2021-02-11T06:11:28.350 INFO:teuthology.orchestra.run.smithi172.stdout:[osd] 2021-02-11T06:11:28.350 INFO:teuthology.orchestra.run.smithi172.stdout: osd journal size = 100 2021-02-11T06:11:28.350 INFO:teuthology.orchestra.run.smithi172.stdout: 2021-02-11T06:11:28.350 INFO:teuthology.orchestra.run.smithi172.stdout: osd scrub load threshold = 5.0 2021-02-11T06:11:28.350 INFO:teuthology.orchestra.run.smithi172.stdout: osd scrub max interval = 600 2021-02-11T06:11:28.350 INFO:teuthology.orchestra.run.smithi172.stdout: 2021-02-11T06:11:28.350 INFO:teuthology.orchestra.run.smithi172.stdout: osd recover clone overlap = true 2021-02-11T06:11:28.350 INFO:teuthology.orchestra.run.smithi172.stdout: osd recovery max chunk = 1048576 2021-02-11T06:11:28.350 INFO:teuthology.orchestra.run.smithi172.stdout: 2021-02-11T06:11:28.350 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug shutdown = true 2021-02-11T06:11:28.350 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug op order = true 2021-02-11T06:11:28.350 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug verify stray on activate = true 2021-02-11T06:11:28.351 INFO:teuthology.orchestra.run.smithi172.stdout: 2021-02-11T06:11:28.351 INFO:teuthology.orchestra.run.smithi172.stdout: osd open classes on start = true 2021-02-11T06:11:28.351 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug pg log writeout = true 2021-02-11T06:11:28.351 INFO:teuthology.orchestra.run.smithi172.stdout: 2021-02-11T06:11:28.351 INFO:teuthology.orchestra.run.smithi172.stdout: osd deep scrub update digest min age = 30 2021-02-11T06:11:28.351 INFO:teuthology.orchestra.run.smithi172.stdout: 2021-02-11T06:11:28.351 INFO:teuthology.orchestra.run.smithi172.stdout: osd map max advance = 10 2021-02-11T06:11:28.351 INFO:teuthology.orchestra.run.smithi172.stdout: 2021-02-11T06:11:28.351 INFO:teuthology.orchestra.run.smithi172.stdout: journal zero on create = true 2021-02-11T06:11:28.351 INFO:teuthology.orchestra.run.smithi172.stdout: 2021-02-11T06:11:28.351 INFO:teuthology.orchestra.run.smithi172.stdout: filestore ondisk finisher threads = 3 2021-02-11T06:11:28.351 INFO:teuthology.orchestra.run.smithi172.stdout: filestore apply finisher threads = 3 2021-02-11T06:11:28.351 INFO:teuthology.orchestra.run.smithi172.stdout: 2021-02-11T06:11:28.351 INFO:teuthology.orchestra.run.smithi172.stdout: bdev debug aio = true 2021-02-11T06:11:28.352 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug misdirected ops = true 2021-02-11T06:11:28.352 INFO:teuthology.orchestra.run.smithi172.stdout: debug filestore = 20 2021-02-11T06:11:28.352 INFO:teuthology.orchestra.run.smithi172.stdout: debug journal = 20 2021-02-11T06:11:28.352 INFO:teuthology.orchestra.run.smithi172.stdout: debug ms = 20 2021-02-11T06:11:28.352 INFO:teuthology.orchestra.run.smithi172.stdout: debug osd = 25 2021-02-11T06:11:28.352 INFO:teuthology.orchestra.run.smithi172.stdout: osd fast shutdown = False 2021-02-11T06:11:28.352 INFO:teuthology.orchestra.run.smithi172.stdout: rocksdb delete range threshold = 0 2021-02-11T06:11:28.352 INFO:teuthology.orchestra.run.smithi172.stdout: 2021-02-11T06:11:28.352 INFO:teuthology.orchestra.run.smithi172.stdout:[mgr] 2021-02-11T06:11:28.352 INFO:teuthology.orchestra.run.smithi172.stdout: debug ms = 1 2021-02-11T06:11:28.352 INFO:teuthology.orchestra.run.smithi172.stdout: debug mgr = 20 2021-02-11T06:11:28.352 INFO:teuthology.orchestra.run.smithi172.stdout: debug mon = 20 2021-02-11T06:11:28.352 INFO:teuthology.orchestra.run.smithi172.stdout: debug auth = 20 2021-02-11T06:11:28.352 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min pgs per osd = 4 2021-02-11T06:11:28.353 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min bytes per osd = 10 2021-02-11T06:11:28.353 INFO:teuthology.orchestra.run.smithi172.stdout: mgr/telemetry/nag = false 2021-02-11T06:11:28.353 INFO:teuthology.orchestra.run.smithi172.stdout: 2021-02-11T06:11:28.353 INFO:teuthology.orchestra.run.smithi172.stdout:[mon] 2021-02-11T06:11:28.353 INFO:teuthology.orchestra.run.smithi172.stdout: debug ms = 1 2021-02-11T06:11:28.353 INFO:teuthology.orchestra.run.smithi172.stdout: debug mon = 20 2021-02-11T06:11:28.353 INFO:teuthology.orchestra.run.smithi172.stdout: debug paxos = 20 2021-02-11T06:11:28.353 INFO:teuthology.orchestra.run.smithi172.stdout: debug auth = 20 2021-02-11T06:11:28.353 INFO:teuthology.orchestra.run.smithi172.stdout: mon data avail warn = 5 2021-02-11T06:11:28.353 INFO:teuthology.orchestra.run.smithi172.stdout: mon mgr mkfs grace = 240 2021-02-11T06:11:28.353 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min pgs per osd = 4 2021-02-11T06:11:28.353 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd reporter subtree level = osd 2021-02-11T06:11:28.353 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd prime pg temp = true 2021-02-11T06:11:28.353 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min bytes per osd = 10 2021-02-11T06:11:28.354 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd crush smoke test = False 2021-02-11T06:11:28.354 INFO:teuthology.orchestra.run.smithi172.stdout: 2021-02-11T06:11:28.354 INFO:teuthology.orchestra.run.smithi172.stdout:[client] 2021-02-11T06:11:28.354 INFO:teuthology.orchestra.run.smithi172.stdout: rgw cache enabled = true 2021-02-11T06:11:28.354 INFO:teuthology.orchestra.run.smithi172.stdout: rgw enable ops log = true 2021-02-11T06:11:28.354 INFO:teuthology.orchestra.run.smithi172.stdout: rgw enable usage log = true 2021-02-11T06:11:28.354 INFO:teuthology.orchestra.run.smithi172.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-02-11T06:11:28.354 INFO:teuthology.orchestra.run.smithi172.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-02-11T06:11:28.354 INFO:teuthology.orchestra.run.smithi172.stdout: debug rgw = 20 2021-02-11T06:11:28.354 INFO:teuthology.orchestra.run.smithi172.stdout: rgw crypt require ssl = False 2021-02-11T06:11:28.354 INFO:teuthology.orchestra.run.smithi172.stdout: rgw crypt s3 kms backend = testing 2021-02-11T06:11:28.354 INFO:teuthology.orchestra.run.smithi172.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-02-11T06:11:28.354 INFO:teuthology.orchestra.run.smithi172.stdout: rgw curl low speed time = 300 2021-02-11T06:11:28.355 INFO:teuthology.orchestra.run.smithi172.stdout: rgw data log num shards = 4 2021-02-11T06:11:28.355 INFO:teuthology.orchestra.run.smithi172.stdout: rgw md log max shards = 4 2021-02-11T06:11:28.355 INFO:teuthology.orchestra.run.smithi172.stdout: rgw sync log trim interval = 0 2021-02-11T06:11:28.355 INFO:teuthology.orchestra.run.smithi172.stdout: rgw sync obj etag verify = True 2021-02-11T06:11:28.355 INFO:teuthology.orchestra.run.smithi172.stdout: setgroup = ceph 2021-02-11T06:11:28.355 INFO:teuthology.orchestra.run.smithi172.stdout: setuser = ceph 2021-02-11T06:11:28.355 INFO:teuthology.orchestra.run.smithi172.stdout:[mon.a] 2021-02-11T06:11:28.355 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.0] 2021-02-11T06:11:28.355 INFO:teuthology.orchestra.run.smithi172.stdout: osd_max_omap_entries_per_request = 10 2021-02-11T06:11:28.355 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.1] 2021-02-11T06:11:28.355 INFO:teuthology.orchestra.run.smithi172.stdout: osd_max_omap_entries_per_request = 1000 2021-02-11T06:11:28.355 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.2] 2021-02-11T06:11:28.355 INFO:teuthology.orchestra.run.smithi172.stdout: osd_max_omap_entries_per_request = 10000 2021-02-11T06:11:28.356 INFO:tasks.ceph:Creating admin key on c2.mon.a... 2021-02-11T06:11:28.357 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c2.keyring 2021-02-11T06:11:28.445 INFO:tasks.ceph:Copying monmap to all nodes... 2021-02-11T06:11:28.445 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2021-02-11T06:11:28.446 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/etc/ceph/c2.keyring of=/dev/stdout 2021-02-11T06:11:28.452 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2021-02-11T06:11:28.452 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/home/ubuntu/cephtest/c2.monmap of=/dev/stdout 2021-02-11T06:11:28.500 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi061.front.sepia.ceph.com 2021-02-11T06:11:28.500 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2021-02-11T06:11:28.500 DEBUG:teuthology.orchestra.run.smithi061:> sudo dd of=/etc/ceph/c2.keyring 2021-02-11T06:11:28.500 DEBUG:teuthology.orchestra.run.smithi061:> sudo chmod 0644 /etc/ceph/c2.keyring 2021-02-11T06:11:28.526 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2021-02-11T06:11:28.526 DEBUG:teuthology.orchestra.run.smithi061:> dd of=/home/ubuntu/cephtest/c2.monmap 2021-02-11T06:11:28.575 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi172.front.sepia.ceph.com 2021-02-11T06:11:28.575 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2021-02-11T06:11:28.575 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/ceph/c2.keyring 2021-02-11T06:11:28.575 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 0644 /etc/ceph/c2.keyring 2021-02-11T06:11:28.600 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2021-02-11T06:11:28.600 DEBUG:teuthology.orchestra.run.smithi172:> dd of=/home/ubuntu/cephtest/c2.monmap 2021-02-11T06:11:28.648 INFO:tasks.ceph:Setting up mon nodes... 2021-02-11T06:11:28.648 INFO:tasks.ceph:Setting up mgr nodes... 2021-02-11T06:11:28.648 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/mgr/c2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c2-x/keyring 2021-02-11T06:11:28.729 INFO:teuthology.orchestra.run.smithi172.stdout:creating /var/lib/ceph/mgr/c2-x/keyring 2021-02-11T06:11:28.731 INFO:tasks.ceph:Setting up mds nodes... 2021-02-11T06:11:28.732 INFO:tasks.ceph_client:Setting up client nodes... 2021-02-11T06:11:28.732 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c2.client.0.keyring && sudo chmod 0644 /etc/ceph/c2.client.0.keyring 2021-02-11T06:11:28.807 INFO:teuthology.orchestra.run.smithi172.stdout:creating /etc/ceph/c2.client.0.keyring 2021-02-11T06:11:28.821 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c2.client.1.keyring && sudo chmod 0644 /etc/ceph/c2.client.1.keyring 2021-02-11T06:11:28.894 INFO:teuthology.orchestra.run.smithi172.stdout:creating /etc/ceph/c2.client.1.keyring 2021-02-11T06:11:28.907 INFO:tasks.ceph:Running mkfs on osd nodes... 2021-02-11T06:11:28.907 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi061.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi172.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2021-02-11T06:11:28.907 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/osd/c2-0 2021-02-11T06:11:28.917 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2021-02-11T06:11:28.917 INFO:tasks.ceph:role: c2.osd.0 2021-02-11T06:11:28.917 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi172.front.sepia.ceph.com 2021-02-11T06:11:28.917 DEBUG:teuthology.orchestra.run.smithi172:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2021-02-11T06:11:29.441 INFO:teuthology.orchestra.run.smithi172.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2021-02-11T06:11:29.442 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 attr=2, projid32bit=1 2021-02-11T06:11:29.442 INFO:teuthology.orchestra.run.smithi172.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2021-02-11T06:11:29.442 INFO:teuthology.orchestra.run.smithi172.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-02-11T06:11:29.442 INFO:teuthology.orchestra.run.smithi172.stdout: = sunit=0 swidth=0 blks 2021-02-11T06:11:29.442 INFO:teuthology.orchestra.run.smithi172.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-02-11T06:11:29.442 INFO:teuthology.orchestra.run.smithi172.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-02-11T06:11:29.442 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-02-11T06:11:29.442 INFO:teuthology.orchestra.run.smithi172.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-02-11T06:11:29.443 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi172.front.sepia.ceph.com -o noatime 2021-02-11T06:11:29.444 DEBUG:teuthology.orchestra.run.smithi172:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c2-0 2021-02-11T06:11:29.518 DEBUG:teuthology.orchestra.run.smithi172:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-0 2021-02-11T06:11:29.527 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: /sbin/restorecon: command not found 2021-02-11T06:11:29.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-11T06:11:29.528 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/osd/c2-1 2021-02-11T06:11:29.585 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2021-02-11T06:11:29.585 INFO:tasks.ceph:role: c2.osd.1 2021-02-11T06:11:29.585 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi172.front.sepia.ceph.com 2021-02-11T06:11:29.586 DEBUG:teuthology.orchestra.run.smithi172:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2021-02-11T06:11:30.091 INFO:teuthology.orchestra.run.smithi172.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2021-02-11T06:11:30.091 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 attr=2, projid32bit=1 2021-02-11T06:11:30.091 INFO:teuthology.orchestra.run.smithi172.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2021-02-11T06:11:30.091 INFO:teuthology.orchestra.run.smithi172.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-02-11T06:11:30.091 INFO:teuthology.orchestra.run.smithi172.stdout: = sunit=0 swidth=0 blks 2021-02-11T06:11:30.091 INFO:teuthology.orchestra.run.smithi172.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-02-11T06:11:30.091 INFO:teuthology.orchestra.run.smithi172.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-02-11T06:11:30.092 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-02-11T06:11:30.092 INFO:teuthology.orchestra.run.smithi172.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-02-11T06:11:30.093 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi172.front.sepia.ceph.com -o noatime 2021-02-11T06:11:30.093 DEBUG:teuthology.orchestra.run.smithi172:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c2-1 2021-02-11T06:11:30.154 DEBUG:teuthology.orchestra.run.smithi172:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-1 2021-02-11T06:11:30.165 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: /sbin/restorecon: command not found 2021-02-11T06:11:30.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-11T06:11:30.165 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/osd/c2-2 2021-02-11T06:11:30.220 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2021-02-11T06:11:30.220 INFO:tasks.ceph:role: c2.osd.2 2021-02-11T06:11:30.221 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi172.front.sepia.ceph.com 2021-02-11T06:11:30.221 DEBUG:teuthology.orchestra.run.smithi172:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2021-02-11T06:11:30.724 INFO:teuthology.orchestra.run.smithi172.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2021-02-11T06:11:30.724 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 attr=2, projid32bit=1 2021-02-11T06:11:30.724 INFO:teuthology.orchestra.run.smithi172.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2021-02-11T06:11:30.724 INFO:teuthology.orchestra.run.smithi172.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-02-11T06:11:30.724 INFO:teuthology.orchestra.run.smithi172.stdout: = sunit=0 swidth=0 blks 2021-02-11T06:11:30.724 INFO:teuthology.orchestra.run.smithi172.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-02-11T06:11:30.724 INFO:teuthology.orchestra.run.smithi172.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-02-11T06:11:30.724 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-02-11T06:11:30.724 INFO:teuthology.orchestra.run.smithi172.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-02-11T06:11:30.726 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi172.front.sepia.ceph.com -o noatime 2021-02-11T06:11:30.726 DEBUG:teuthology.orchestra.run.smithi172:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c2-2 2021-02-11T06:11:30.772 DEBUG:teuthology.orchestra.run.smithi172:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-2 2021-02-11T06:11:30.782 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: /sbin/restorecon: command not found 2021-02-11T06:11:30.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-11T06:11:30.783 DEBUG:teuthology.orchestra.run.smithi172:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c2.monmap 2021-02-11T06:11:30.877 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:11:30.877+0000 7f4c78137080 -1 auth: error reading file: /var/lib/ceph/osd/c2-0/keyring: can't open /var/lib/ceph/osd/c2-0/keyring: (2) No such file or directory 2021-02-11T06:11:30.879 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:11:30.881+0000 7f4c78137080 -1 created new key in keyring /var/lib/ceph/osd/c2-0/keyring 2021-02-11T06:11:30.880 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:11:30.881+0000 7f4c78137080 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2021-02-11T06:11:30.880 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:11:30.881+0000 7f4c78137080 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2021-02-11T06:11:30.880 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:11:30.881+0000 7f4c78137080 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2021-02-11T06:11:30.883 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:11:30.885+0000 7f4c78137080 -1 bluestore(/var/lib/ceph/osd/c2-0) _read_fsid unparsable uuid 2021-02-11T06:11:33.040 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-0 2021-02-11T06:11:43.260 DEBUG:teuthology.orchestra.run.smithi172:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c2.monmap 2021-02-11T06:11:43.338 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:11:43.336+0000 7f58edc35080 -1 auth: error reading file: /var/lib/ceph/osd/c2-1/keyring: can't open /var/lib/ceph/osd/c2-1/keyring: (2) No such file or directory 2021-02-11T06:11:43.338 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:11:43.336+0000 7f58edc35080 -1 created new key in keyring /var/lib/ceph/osd/c2-1/keyring 2021-02-11T06:11:43.338 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:11:43.336+0000 7f58edc35080 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2021-02-11T06:11:43.338 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:11:43.336+0000 7f58edc35080 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2021-02-11T06:11:43.338 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:11:43.336+0000 7f58edc35080 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2021-02-11T06:11:43.339 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:11:43.336+0000 7f58edc35080 -1 bluestore(/var/lib/ceph/osd/c2-1) _read_fsid unparsable uuid 2021-02-11T06:11:45.512 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-1 2021-02-11T06:11:45.571 DEBUG:teuthology.orchestra.run.smithi172:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c2.monmap 2021-02-11T06:11:45.648 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:11:45.648+0000 7f33437bd080 -1 auth: error reading file: /var/lib/ceph/osd/c2-2/keyring: can't open /var/lib/ceph/osd/c2-2/keyring: (2) No such file or directory 2021-02-11T06:11:45.649 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:11:45.648+0000 7f33437bd080 -1 created new key in keyring /var/lib/ceph/osd/c2-2/keyring 2021-02-11T06:11:45.649 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:11:45.648+0000 7f33437bd080 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2021-02-11T06:11:45.649 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:11:45.648+0000 7f33437bd080 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2021-02-11T06:11:45.649 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:11:45.648+0000 7f33437bd080 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2021-02-11T06:11:45.649 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:11:45.648+0000 7f33437bd080 -1 bluestore(/var/lib/ceph/osd/c2-2) _read_fsid unparsable uuid 2021-02-11T06:11:47.811 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-2 2021-02-11T06:11:47.867 INFO:tasks.ceph:Reading keys from all nodes... 2021-02-11T06:11:47.867 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2021-02-11T06:11:47.868 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/mgr/c2-x/keyring of=/dev/stdout 2021-02-11T06:11:47.922 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2021-02-11T06:11:47.922 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/c2-0/keyring of=/dev/stdout 2021-02-11T06:11:47.978 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2021-02-11T06:11:47.978 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/c2-1/keyring of=/dev/stdout 2021-02-11T06:11:48.036 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2021-02-11T06:11:48.036 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/c2-2/keyring of=/dev/stdout 2021-02-11T06:11:48.090 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2021-02-11T06:11:48.090 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/etc/ceph/c2.client.0.keyring of=/dev/stdout 2021-02-11T06:11:48.138 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2021-02-11T06:11:48.138 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/etc/ceph/c2.client.1.keyring of=/dev/stdout 2021-02-11T06:11:48.185 INFO:tasks.ceph:Adding keys to all mons... 2021-02-11T06:11:48.185 DEBUG:teuthology.orchestra.run.smithi172:> sudo tee -a /etc/ceph/c2.keyring 2021-02-11T06:11:48.241 INFO:teuthology.orchestra.run.smithi172.stdout:[mgr.x] 2021-02-11T06:11:48.241 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQCQyiRg7ynJKxAAo5GKvVNvZMUwY03AhBp0sA== 2021-02-11T06:11:48.241 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.0] 2021-02-11T06:11:48.241 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQCSyiRgWByNNBAAJiMWHwlzgAMgb5b+dVdYOw== 2021-02-11T06:11:48.241 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.1] 2021-02-11T06:11:48.241 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQCfyiRgJA1UFBAAV6byNJBoNJfE7ADD2B3g0A== 2021-02-11T06:11:48.241 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.2] 2021-02-11T06:11:48.242 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQChyiRgeN7aJhAAv/F7CFalOq2//nkQuqKUzA== 2021-02-11T06:11:48.242 INFO:teuthology.orchestra.run.smithi172.stdout:[client.0] 2021-02-11T06:11:48.242 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQCQyiRglktsMBAAXPhyMJAMu60zrW6Vw5oMCQ== 2021-02-11T06:11:48.242 INFO:teuthology.orchestra.run.smithi172.stdout:[client.1] 2021-02-11T06:11:48.242 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQCQyiRg7sScNRAAPna+F8OHoGkQQTdQ3bs89Q== 2021-02-11T06:11:48.243 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2021-02-11T06:11:48.324 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-11T06:11:48.410 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-11T06:11:48.494 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-11T06:11:48.577 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-02-11T06:11:48.664 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-02-11T06:11:48.751 INFO:tasks.ceph:Running mkfs on mon nodes... 2021-02-11T06:11:48.751 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/mon/c2-a 2021-02-11T06:11:48.805 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c2 --mkfs -i a --monmap /home/ubuntu/cephtest/c2.monmap --keyring /etc/ceph/c2.keyring 2021-02-11T06:11:48.906 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c2-a 2021-02-11T06:11:48.916 DEBUG:teuthology.orchestra.run.smithi172:> rm -- /home/ubuntu/cephtest/c2.monmap 2021-02-11T06:11:48.965 INFO:tasks.ceph:Starting mon daemons in cluster c2... 2021-02-11T06:11:48.966 INFO:tasks.ceph.c2.mon.a:Restarting daemon 2021-02-11T06:11:48.966 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c2 -i a 2021-02-11T06:11:49.009 INFO:tasks.ceph.c2.mon.a:Started 2021-02-11T06:11:49.009 INFO:tasks.ceph:Starting mgr daemons in cluster c2... 2021-02-11T06:11:49.010 INFO:tasks.ceph.c2.mgr.x:Restarting daemon 2021-02-11T06:11:49.010 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c2 -i x 2021-02-11T06:11:49.012 INFO:tasks.ceph.c2.mgr.x:Started 2021-02-11T06:11:49.012 INFO:tasks.ceph:Setting crush tunables to default 2021-02-11T06:11:49.012 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster c2 osd crush tunables default 2021-02-11T06:11:49.419 INFO:teuthology.orchestra.run.smithi172.stderr:adjusted tunables profile to default 2021-02-11T06:11:49.430 INFO:tasks.ceph:Starting osd daemons in cluster c2... 2021-02-11T06:11:49.430 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2021-02-11T06:11:49.430 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/c2-0/fsid of=/dev/stdout 2021-02-11T06:11:49.440 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2021-02-11T06:11:49.440 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/c2-1/fsid of=/dev/stdout 2021-02-11T06:11:49.496 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2021-02-11T06:11:49.496 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/c2-2/fsid of=/dev/stdout 2021-02-11T06:11:49.553 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster c2 osd new c5cdebd7-634e-47c8-98cb-ab4dfea1e594 0 2021-02-11T06:11:49.813 INFO:teuthology.orchestra.run.smithi172.stdout:0 2021-02-11T06:11:49.827 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster c2 osd new 85ae3f4d-8dd9-4363-8a1f-d31d71884fe8 1 2021-02-11T06:11:50.043 INFO:teuthology.orchestra.run.smithi172.stdout:1 2021-02-11T06:11:50.056 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster c2 osd new 3a60e445-b4ef-42ea-8328-496da35e31b7 2 2021-02-11T06:11:50.265 INFO:teuthology.orchestra.run.smithi172.stdout:2 2021-02-11T06:11:50.278 INFO:tasks.ceph.c2.osd.0:Restarting daemon 2021-02-11T06:11:50.278 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 0 2021-02-11T06:11:50.281 INFO:tasks.ceph.c2.osd.0:Started 2021-02-11T06:11:50.281 INFO:tasks.ceph.c2.osd.1:Restarting daemon 2021-02-11T06:11:50.281 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 1 2021-02-11T06:11:50.283 INFO:tasks.ceph.c2.osd.1:Started 2021-02-11T06:11:50.283 INFO:tasks.ceph.c2.osd.2:Restarting daemon 2021-02-11T06:11:50.283 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 2 2021-02-11T06:11:50.285 INFO:tasks.ceph.c2.osd.2:Started 2021-02-11T06:11:50.285 INFO:tasks.ceph:Waiting for OSDs to come up 2021-02-11T06:11:50.588 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2021-02-11T06:11:50.779 INFO:teuthology.misc.health.smithi172.stdout: 2021-02-11T06:11:50.779 INFO:teuthology.misc.health.smithi172.stdout:{"epoch":5,"fsid":"0d6e87fb-daa0-4c85-a370-bf71998465da","created":"2021-02-11T06:11:49.134481+0000","modified":"2021-02-11T06:11:50.266213+0000","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"c5cdebd7-634e-47c8-98cb-ab4dfea1e594","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"85ae3f4d-8dd9-4363-8a1f-d31d71884fe8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3a60e445-b4ef-42ea-8328-496da35e31b7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-02-11T06:11:50.790 DEBUG:teuthology.misc:0 of 3 OSDs are up 2021-02-11T06:11:50.894 INFO:tasks.ceph.c2.osd.0.smithi172.stderr:2021-02-11T06:11:50.892+0000 7f18af4e2080 -1 Falling back to public interface 2021-02-11T06:11:50.897 INFO:tasks.ceph.c2.osd.1.smithi172.stderr:2021-02-11T06:11:50.896+0000 7fd69deba080 -1 Falling back to public interface 2021-02-11T06:11:50.898 INFO:tasks.ceph.c2.osd.2.smithi172.stderr:2021-02-11T06:11:50.896+0000 7f4e55b17080 -1 Falling back to public interface 2021-02-11T06:11:53.830 INFO:tasks.ceph.c2.osd.1.smithi172.stderr:2021-02-11T06:11:53.828+0000 7fd69deba080 -1 osd.1 0 log_to_monitors {default=true} 2021-02-11T06:12:03.417 INFO:tasks.ceph.c2.osd.2.smithi172.stderr:2021-02-11T06:11:53.888+0000 7f4e55b17080 -1 osd.2 0 log_to_monitors {default=true} 2021-02-11T06:12:03.417 INFO:tasks.ceph.c2.osd.2.smithi172.stderr:2021-02-11T06:11:55.912+0000 7f4e4cfe0700 -1 osd.2 0 waiting for initial osdmap 2021-02-11T06:12:03.418 INFO:tasks.ceph.c2.osd.0.smithi172.stderr:2021-02-11T06:11:53.912+0000 7f18af4e2080 -1 osd.0 0 log_to_monitors {default=true} 2021-02-11T06:12:03.418 INFO:tasks.ceph.c2.osd.0.smithi172.stderr:2021-02-11T06:11:55.912+0000 7f18a81ae700 -1 osd.0 0 waiting for initial osdmap 2021-02-11T06:12:03.418 INFO:tasks.ceph.c2.osd.1.smithi172.stderr:2021-02-11T06:11:54.904+0000 7fd695383700 -1 osd.1 0 waiting for initial osdmap 2021-02-11T06:12:03.718 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2021-02-11T06:12:04.013 INFO:teuthology.misc.health.smithi172.stdout: 2021-02-11T06:12:04.014 INFO:teuthology.misc.health.smithi172.stdout:{"epoch":12,"fsid":"0d6e87fb-daa0-4c85-a370-bf71998465da","created":"2021-02-11T06:11:49.134481+0000","modified":"2021-02-11T06:11:59.937255+0000","last_up_change":"2021-02-11T06:11:56.924613+0000","last_in_change":"2021-02-11T06:11:56.924613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-02-11T06:11:56.930707+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"c5cdebd7-634e-47c8-98cb-ab4dfea1e594","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":14675},{"type":"v1","addr":"172.21.15.172:6801","nonce":14675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":14675},{"type":"v1","addr":"172.21.15.172:6803","nonce":14675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":14675},{"type":"v1","addr":"172.21.15.172:6807","nonce":14675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":14675},{"type":"v1","addr":"172.21.15.172:6805","nonce":14675}]},"public_addr":"172.21.15.172:6801/14675","cluster_addr":"172.21.15.172:6803/14675","heartbeat_back_addr":"172.21.15.172:6807/14675","heartbeat_front_addr":"172.21.15.172:6805/14675","state":["exists","up"]},{"osd":1,"uuid":"85ae3f4d-8dd9-4363-8a1f-d31d71884fe8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":14677},{"type":"v1","addr":"172.21.15.172:6809","nonce":14677}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":14677},{"type":"v1","addr":"172.21.15.172:6812","nonce":14677}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":14677},{"type":"v1","addr":"172.21.15.172:6819","nonce":14677}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":14677},{"type":"v1","addr":"172.21.15.172:6815","nonce":14677}]},"public_addr":"172.21.15.172:6809/14677","cluster_addr":"172.21.15.172:6812/14677","heartbeat_back_addr":"172.21.15.172:6819/14677","heartbeat_front_addr":"172.21.15.172:6815/14677","state":["exists","up"]},{"osd":2,"uuid":"3a60e445-b4ef-42ea-8328-496da35e31b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6811","nonce":14676},{"type":"v1","addr":"172.21.15.172:6813","nonce":14676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":14676},{"type":"v1","addr":"172.21.15.172:6818","nonce":14676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":14676},{"type":"v1","addr":"172.21.15.172:6823","nonce":14676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":14676},{"type":"v1","addr":"172.21.15.172:6821","nonce":14676}]},"public_addr":"172.21.15.172:6813/14676","cluster_addr":"172.21.15.172:6818/14676","heartbeat_back_addr":"172.21.15.172:6823/14676","heartbeat_front_addr":"172.21.15.172:6821/14676","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T06:11:54.944094+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T06:11:54.812675+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T06:11:54.941007+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-02-11T06:12:04.028 DEBUG:teuthology.misc:3 of 3 OSDs are up 2021-02-11T06:12:04.028 INFO:tasks.ceph:Creating RBD pool 2021-02-11T06:12:04.028 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster c2 osd pool create rbd 8 2021-02-11T06:12:05.116 INFO:teuthology.orchestra.run.smithi172.stderr:pool 'rbd' created 2021-02-11T06:12:05.130 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster c2 osd pool application enable rbd rbd --yes-i-really-mean-it 2021-02-11T06:12:06.123 INFO:teuthology.orchestra.run.smithi172.stderr:enabled application 'rbd' on pool 'rbd' 2021-02-11T06:12:06.140 INFO:tasks.ceph:Starting mds daemons in cluster c2... 2021-02-11T06:12:06.141 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2021-02-11T06:12:06.141 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2021-02-11T06:12:06.434 INFO:teuthology.orchestra.run.smithi172.stdout: 2021-02-11T06:12:06.434 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":14,"fsid":"0d6e87fb-daa0-4c85-a370-bf71998465da","created":"2021-02-11T06:11:49.134481+0000","modified":"2021-02-11T06:12:06.107178+0000","last_up_change":"2021-02-11T06:11:56.924613+0000","last_in_change":"2021-02-11T06:11:56.924613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-02-11T06:11:56.930707+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-02-11T06:12:04.328307+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"c5cdebd7-634e-47c8-98cb-ab4dfea1e594","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":14675},{"type":"v1","addr":"172.21.15.172:6801","nonce":14675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":14675},{"type":"v1","addr":"172.21.15.172:6803","nonce":14675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":14675},{"type":"v1","addr":"172.21.15.172:6807","nonce":14675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":14675},{"type":"v1","addr":"172.21.15.172:6805","nonce":14675}]},"public_addr":"172.21.15.172:6801/14675","cluster_addr":"172.21.15.172:6803/14675","heartbeat_back_addr":"172.21.15.172:6807/14675","heartbeat_front_addr":"172.21.15.172:6805/14675","state":["exists","up"]},{"osd":1,"uuid":"85ae3f4d-8dd9-4363-8a1f-d31d71884fe8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":14677},{"type":"v1","addr":"172.21.15.172:6809","nonce":14677}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":14677},{"type":"v1","addr":"172.21.15.172:6812","nonce":14677}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":14677},{"type":"v1","addr":"172.21.15.172:6819","nonce":14677}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":14677},{"type":"v1","addr":"172.21.15.172:6815","nonce":14677}]},"public_addr":"172.21.15.172:6809/14677","cluster_addr":"172.21.15.172:6812/14677","heartbeat_back_addr":"172.21.15.172:6819/14677","heartbeat_front_addr":"172.21.15.172:6815/14677","state":["exists","up"]},{"osd":2,"uuid":"3a60e445-b4ef-42ea-8328-496da35e31b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6811","nonce":14676},{"type":"v1","addr":"172.21.15.172:6813","nonce":14676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":14676},{"type":"v1","addr":"172.21.15.172:6818","nonce":14676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":14676},{"type":"v1","addr":"172.21.15.172:6823","nonce":14676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":14676},{"type":"v1","addr":"172.21.15.172:6821","nonce":14676}]},"public_addr":"172.21.15.172:6813/14676","cluster_addr":"172.21.15.172:6818/14676","heartbeat_back_addr":"172.21.15.172:6823/14676","heartbeat_front_addr":"172.21.15.172:6821/14676","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T06:11:54.944094+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T06:11:54.812675+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T06:11:54.941007+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-02-11T06:12:06.448 INFO:tasks.ceph.ceph_manager.c2:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2021-02-11T06:11:56.930707+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '12', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2021-02-11T06:12:04.328307+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '14', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-11T06:12:06.448 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd pool get device_health_metrics pg_num 2021-02-11T06:12:06.744 INFO:teuthology.orchestra.run.smithi172.stdout:pg_num: 1 2021-02-11T06:12:06.758 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd pool get rbd pg_num 2021-02-11T06:12:07.061 INFO:teuthology.orchestra.run.smithi172.stdout:pg_num: 8 2021-02-11T06:12:07.074 INFO:tasks.ceph:Waiting until c2 daemons up and pgs clean... 2021-02-11T06:12:07.074 INFO:tasks.ceph.ceph_manager.c2:waiting for mgr available 2021-02-11T06:12:07.075 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 mgr dump --format=json 2021-02-11T06:12:07.404 INFO:teuthology.orchestra.run.smithi172.stdout: 2021-02-11T06:12:07.424 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":4,"active_gid":4098,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":14492},{"type":"v1","addr":"172.21.15.172:6825","nonce":14492}]},"active_addr":"172.21.15.172:6825/14492","active_change":"2021-02-11T06:11:53.912601+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading remoto library:No module named 'remoto'","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"haproxy","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.172:0","nonce":1397385250}]},{"addrvec":[{"type":"v2","addr":"172.21.15.172:0","nonce":28174985}]}]}} 2021-02-11T06:12:07.426 INFO:tasks.ceph.ceph_manager.c2:mgr available! 2021-02-11T06:12:07.426 INFO:tasks.ceph.ceph_manager.c2:waiting for all up 2021-02-11T06:12:07.426 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2021-02-11T06:12:07.720 INFO:teuthology.orchestra.run.smithi172.stdout: 2021-02-11T06:12:08.450 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":15,"fsid":"0d6e87fb-daa0-4c85-a370-bf71998465da","created":"2021-02-11T06:11:49.134481+0000","modified":"2021-02-11T06:12:07.113206+0000","last_up_change":"2021-02-11T06:11:56.924613+0000","last_in_change":"2021-02-11T06:11:56.924613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-02-11T06:11:56.930707+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-02-11T06:12:04.328307+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"c5cdebd7-634e-47c8-98cb-ab4dfea1e594","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":14675},{"type":"v1","addr":"172.21.15.172:6801","nonce":14675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":14675},{"type":"v1","addr":"172.21.15.172:6803","nonce":14675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":14675},{"type":"v1","addr":"172.21.15.172:6807","nonce":14675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":14675},{"type":"v1","addr":"172.21.15.172:6805","nonce":14675}]},"public_addr":"172.21.15.172:6801/14675","cluster_addr":"172.21.15.172:6803/14675","heartbeat_back_addr":"172.21.15.172:6807/14675","heartbeat_front_addr":"172.21.15.172:6805/14675","state":["exists","up"]},{"osd":1,"uuid":"85ae3f4d-8dd9-4363-8a1f-d31d71884fe8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":14677},{"type":"v1","addr":"172.21.15.172:6809","nonce":14677}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":14677},{"type":"v1","addr":"172.21.15.172:6812","nonce":14677}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":14677},{"type":"v1","addr":"172.21.15.172:6819","nonce":14677}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":14677},{"type":"v1","addr":"172.21.15.172:6815","nonce":14677}]},"public_addr":"172.21.15.172:6809/14677","cluster_addr":"172.21.15.172:6812/14677","heartbeat_back_addr":"172.21.15.172:6819/14677","heartbeat_front_addr":"172.21.15.172:6815/14677","state":["exists","up"]},{"osd":2,"uuid":"3a60e445-b4ef-42ea-8328-496da35e31b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6811","nonce":14676},{"type":"v1","addr":"172.21.15.172:6813","nonce":14676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":14676},{"type":"v1","addr":"172.21.15.172:6818","nonce":14676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":14676},{"type":"v1","addr":"172.21.15.172:6823","nonce":14676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":14676},{"type":"v1","addr":"172.21.15.172:6821","nonce":14676}]},"public_addr":"172.21.15.172:6813/14676","cluster_addr":"172.21.15.172:6818/14676","heartbeat_back_addr":"172.21.15.172:6823/14676","heartbeat_front_addr":"172.21.15.172:6821/14676","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T06:11:54.944094+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T06:11:54.812675+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T06:11:54.941007+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-02-11T06:12:08.452 INFO:tasks.ceph.ceph_manager.c2:all up! 2021-02-11T06:12:08.452 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2021-02-11T06:12:08.760 INFO:teuthology.orchestra.run.smithi172.stdout: 2021-02-11T06:12:08.760 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":15,"fsid":"0d6e87fb-daa0-4c85-a370-bf71998465da","created":"2021-02-11T06:11:49.134481+0000","modified":"2021-02-11T06:12:07.113206+0000","last_up_change":"2021-02-11T06:11:56.924613+0000","last_in_change":"2021-02-11T06:11:56.924613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-02-11T06:11:56.930707+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-02-11T06:12:04.328307+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"c5cdebd7-634e-47c8-98cb-ab4dfea1e594","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":14675},{"type":"v1","addr":"172.21.15.172:6801","nonce":14675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":14675},{"type":"v1","addr":"172.21.15.172:6803","nonce":14675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":14675},{"type":"v1","addr":"172.21.15.172:6807","nonce":14675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":14675},{"type":"v1","addr":"172.21.15.172:6805","nonce":14675}]},"public_addr":"172.21.15.172:6801/14675","cluster_addr":"172.21.15.172:6803/14675","heartbeat_back_addr":"172.21.15.172:6807/14675","heartbeat_front_addr":"172.21.15.172:6805/14675","state":["exists","up"]},{"osd":1,"uuid":"85ae3f4d-8dd9-4363-8a1f-d31d71884fe8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":14677},{"type":"v1","addr":"172.21.15.172:6809","nonce":14677}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":14677},{"type":"v1","addr":"172.21.15.172:6812","nonce":14677}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":14677},{"type":"v1","addr":"172.21.15.172:6819","nonce":14677}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":14677},{"type":"v1","addr":"172.21.15.172:6815","nonce":14677}]},"public_addr":"172.21.15.172:6809/14677","cluster_addr":"172.21.15.172:6812/14677","heartbeat_back_addr":"172.21.15.172:6819/14677","heartbeat_front_addr":"172.21.15.172:6815/14677","state":["exists","up"]},{"osd":2,"uuid":"3a60e445-b4ef-42ea-8328-496da35e31b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6811","nonce":14676},{"type":"v1","addr":"172.21.15.172:6813","nonce":14676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":14676},{"type":"v1","addr":"172.21.15.172:6818","nonce":14676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":14676},{"type":"v1","addr":"172.21.15.172:6823","nonce":14676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":14676},{"type":"v1","addr":"172.21.15.172:6821","nonce":14676}]},"public_addr":"172.21.15.172:6813/14676","cluster_addr":"172.21.15.172:6818/14676","heartbeat_back_addr":"172.21.15.172:6823/14676","heartbeat_front_addr":"172.21.15.172:6821/14676","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T06:11:54.944094+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T06:11:54.812675+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T06:11:54.941007+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-02-11T06:12:08.776 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.0 flush_pg_stats 2021-02-11T06:12:08.943 INFO:teuthology.orchestra.run.smithi172.stdout:38654705670 2021-02-11T06:12:08.944 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.1 flush_pg_stats 2021-02-11T06:12:09.112 INFO:teuthology.orchestra.run.smithi172.stdout:34359738373 2021-02-11T06:12:09.113 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.2 flush_pg_stats 2021-02-11T06:12:09.276 INFO:teuthology.orchestra.run.smithi172.stdout:38654705670 2021-02-11T06:12:09.276 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2021-02-11T06:12:09.577 INFO:teuthology.orchestra.run.smithi172.stdout:38654705669 2021-02-11T06:12:09.591 INFO:tasks.ceph.ceph_manager.c2:need seq 38654705670 got 38654705669 for osd.0 2021-02-11T06:12:10.592 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2021-02-11T06:12:10.903 INFO:teuthology.orchestra.run.smithi172.stdout:38654705670 2021-02-11T06:12:10.935 INFO:tasks.ceph.ceph_manager.c2:need seq 38654705670 got 38654705670 for osd.0 2021-02-11T06:12:10.935 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2021-02-11T06:12:11.233 INFO:teuthology.orchestra.run.smithi172.stdout:34359738373 2021-02-11T06:12:11.247 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738373 got 34359738373 for osd.1 2021-02-11T06:12:11.247 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2021-02-11T06:12:11.540 INFO:teuthology.orchestra.run.smithi172.stdout:38654705670 2021-02-11T06:12:11.560 INFO:tasks.ceph.ceph_manager.c2:need seq 38654705670 got 38654705670 for osd.2 2021-02-11T06:12:11.560 INFO:tasks.ceph.ceph_manager.c2:waiting for clean 2021-02-11T06:12:11.560 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2021-02-11T06:12:11.852 INFO:teuthology.orchestra.run.smithi172.stdout: 2021-02-11T06:12:11.853 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2021-02-11T06:12:11.867 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2021-02-11T06:12:09.920756+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":7,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":14792,"kb_used_data":368,"kb_used_omap":0,"kb_used_meta":14400,"kb_avail":314558008,"statfs":{"total":322122547200,"available":322107400192,"internally_reserved":0,"allocated":376832,"data_stored":105519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":14745600},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"8","reported_epoch":"14","state":"active+clean","last_fresh":"2021-02-11T06:12:06.129501+0000","last_change":"2021-02-11T06:12:06.129501+0000","last_active":"2021-02-11T06:12:06.129501+0000","last_peered":"2021-02-11T06:12:06.129501+0000","last_clean":"2021-02-11T06:12:06.129501+0000","last_became_active":"2021-02-11T06:12:06.129164+0000","last_became_peered":"2021-02-11T06:12:06.129164+0000","last_unstale":"2021-02-11T06:12:06.129501+0000","last_undegraded":"2021-02-11T06:12:06.129501+0000","last_fullsized":"2021-02-11T06:12:06.129501+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T06:12:05.104971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T06:12:05.104971+0000","last_clean_scrub_stamp":"2021-02-11T06:12:05.104971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"8","reported_epoch":"14","state":"active+clean","last_fresh":"2021-02-11T06:12:06.129979+0000","last_change":"2021-02-11T06:12:06.129979+0000","last_active":"2021-02-11T06:12:06.129979+0000","last_peered":"2021-02-11T06:12:06.129979+0000","last_clean":"2021-02-11T06:12:06.129979+0000","last_became_active":"2021-02-11T06:12:06.129748+0000","last_became_peered":"2021-02-11T06:12:06.129748+0000","last_unstale":"2021-02-11T06:12:06.129979+0000","last_undegraded":"2021-02-11T06:12:06.129979+0000","last_fullsized":"2021-02-11T06:12:06.129979+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T06:12:05.104971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T06:12:05.104971+0000","last_clean_scrub_stamp":"2021-02-11T06:12:05.104971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"8","reported_epoch":"14","state":"active+clean","last_fresh":"2021-02-11T06:12:06.129472+0000","last_change":"2021-02-11T06:12:06.129472+0000","last_active":"2021-02-11T06:12:06.129472+0000","last_peered":"2021-02-11T06:12:06.129472+0000","last_clean":"2021-02-11T06:12:06.129472+0000","last_became_active":"2021-02-11T06:12:06.129032+0000","last_became_peered":"2021-02-11T06:12:06.129032+0000","last_unstale":"2021-02-11T06:12:06.129472+0000","last_undegraded":"2021-02-11T06:12:06.129472+0000","last_fullsized":"2021-02-11T06:12:06.129472+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T06:12:05.104971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T06:12:05.104971+0000","last_clean_scrub_stamp":"2021-02-11T06:12:05.104971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"8","reported_epoch":"14","state":"active+clean","last_fresh":"2021-02-11T06:12:06.135316+0000","last_change":"2021-02-11T06:12:06.135316+0000","last_active":"2021-02-11T06:12:06.135316+0000","last_peered":"2021-02-11T06:12:06.135316+0000","last_clean":"2021-02-11T06:12:06.135316+0000","last_became_active":"2021-02-11T06:12:06.134933+0000","last_became_peered":"2021-02-11T06:12:06.134933+0000","last_unstale":"2021-02-11T06:12:06.135316+0000","last_undegraded":"2021-02-11T06:12:06.135316+0000","last_fullsized":"2021-02-11T06:12:06.135316+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T06:12:05.104971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T06:12:05.104971+0000","last_clean_scrub_stamp":"2021-02-11T06:12:05.104971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"11","reported_epoch":"13","state":"active+clean","last_fresh":"2021-02-11T06:12:05.109955+0000","last_change":"2021-02-11T06:11:58.970814+0000","last_active":"2021-02-11T06:12:05.109955+0000","last_peered":"2021-02-11T06:12:05.109955+0000","last_clean":"2021-02-11T06:12:05.109955+0000","last_became_active":"2021-02-11T06:11:58.969724+0000","last_became_peered":"2021-02-11T06:11:58.969724+0000","last_unstale":"2021-02-11T06:12:05.109955+0000","last_undegraded":"2021-02-11T06:12:05.109955+0000","last_fullsized":"2021-02-11T06:12:05.109955+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T06:11:57.925734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T06:11:57.925734+0000","last_clean_scrub_stamp":"2021-02-11T06:11:57.925734+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"8","reported_epoch":"14","state":"active+clean","last_fresh":"2021-02-11T06:12:06.132507+0000","last_change":"2021-02-11T06:12:06.132507+0000","last_active":"2021-02-11T06:12:06.132507+0000","last_peered":"2021-02-11T06:12:06.132507+0000","last_clean":"2021-02-11T06:12:06.132507+0000","last_became_active":"2021-02-11T06:12:06.132084+0000","last_became_peered":"2021-02-11T06:12:06.132084+0000","last_unstale":"2021-02-11T06:12:06.132507+0000","last_undegraded":"2021-02-11T06:12:06.132507+0000","last_fullsized":"2021-02-11T06:12:06.132507+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T06:12:05.104971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T06:12:05.104971+0000","last_clean_scrub_stamp":"2021-02-11T06:12:05.104971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"8","reported_epoch":"14","state":"active+clean","last_fresh":"2021-02-11T06:12:06.135133+0000","last_change":"2021-02-11T06:12:06.135133+0000","last_active":"2021-02-11T06:12:06.135133+0000","last_peered":"2021-02-11T06:12:06.135133+0000","last_clean":"2021-02-11T06:12:06.135133+0000","last_became_active":"2021-02-11T06:12:06.134730+0000","last_became_peered":"2021-02-11T06:12:06.134730+0000","last_unstale":"2021-02-11T06:12:06.135133+0000","last_undegraded":"2021-02-11T06:12:06.135133+0000","last_fullsized":"2021-02-11T06:12:06.135133+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T06:12:05.104971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T06:12:05.104971+0000","last_clean_scrub_stamp":"2021-02-11T06:12:05.104971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2021-02-11T06:12:06.136771+0000","last_change":"2021-02-11T06:12:06.132913+0000","last_active":"2021-02-11T06:12:06.136771+0000","last_peered":"2021-02-11T06:12:06.136771+0000","last_clean":"2021-02-11T06:12:06.136771+0000","last_became_active":"2021-02-11T06:12:06.132478+0000","last_became_peered":"2021-02-11T06:12:06.132478+0000","last_unstale":"2021-02-11T06:12:06.136771+0000","last_undegraded":"2021-02-11T06:12:06.136771+0000","last_fullsized":"2021-02-11T06:12:06.136771+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T06:12:05.104971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T06:12:05.104971+0000","last_clean_scrub_stamp":"2021-02-11T06:12:05.104971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"8","reported_epoch":"14","state":"active+clean","last_fresh":"2021-02-11T06:12:06.129927+0000","last_change":"2021-02-11T06:12:06.129927+0000","last_active":"2021-02-11T06:12:06.129927+0000","last_peered":"2021-02-11T06:12:06.129927+0000","last_clean":"2021-02-11T06:12:06.129927+0000","last_became_active":"2021-02-11T06:12:06.129535+0000","last_became_peered":"2021-02-11T06:12:06.129535+0000","last_unstale":"2021-02-11T06:12:06.129927+0000","last_undegraded":"2021-02-11T06:12:06.129927+0000","last_fullsized":"2021-02-11T06:12:06.129927+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T06:12:05.104971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T06:12:05.104971+0000","last_clean_scrub_stamp":"2021-02-11T06:12:05.104971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4928,"kb_used_data":120,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852672,"statfs":{"total":107374182400,"available":107369136128,"internally_reserved":0,"allocated":122880,"data_stored":34002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.177}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.262}]}]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4928,"kb_used_data":120,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852672,"statfs":{"total":107374182400,"available":107369136128,"internally_reserved":0,"allocated":122880,"data_stored":34002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0649999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82699999999999996}]}]},{"osd":0,"up_from":9,"seq":38654705670,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4936,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852664,"statfs":{"total":107374182400,"available":107369127936,"internally_reserved":0,"allocated":131072,"data_stored":37515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.294}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.212}]}]}]}} 2021-02-11T06:12:16.063 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2021-02-11T06:12:16.361 INFO:teuthology.orchestra.run.smithi172.stdout: 2021-02-11T06:12:16.363 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2021-02-11T06:12:16.375 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2021-02-11T06:12:15.922279+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":14808,"kb_used_data":384,"kb_used_omap":0,"kb_used_meta":14400,"kb_avail":314557992,"statfs":{"total":322122547200,"available":322107383808,"internally_reserved":0,"allocated":393216,"data_stored":112545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":14745600},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2021-02-11T06:12:06.134271+0000","last_change":"2021-02-11T06:12:06.129501+0000","last_active":"2021-02-11T06:12:06.134271+0000","last_peered":"2021-02-11T06:12:06.134271+0000","last_clean":"2021-02-11T06:12:06.134271+0000","last_became_active":"2021-02-11T06:12:06.129164+0000","last_became_peered":"2021-02-11T06:12:06.129164+0000","last_unstale":"2021-02-11T06:12:06.134271+0000","last_undegraded":"2021-02-11T06:12:06.134271+0000","last_fullsized":"2021-02-11T06:12:06.134271+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T06:12:05.104971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T06:12:05.104971+0000","last_clean_scrub_stamp":"2021-02-11T06:12:05.104971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2021-02-11T06:12:06.134293+0000","last_change":"2021-02-11T06:12:06.129979+0000","last_active":"2021-02-11T06:12:06.134293+0000","last_peered":"2021-02-11T06:12:06.134293+0000","last_clean":"2021-02-11T06:12:06.134293+0000","last_became_active":"2021-02-11T06:12:06.129748+0000","last_became_peered":"2021-02-11T06:12:06.129748+0000","last_unstale":"2021-02-11T06:12:06.134293+0000","last_undegraded":"2021-02-11T06:12:06.134293+0000","last_fullsized":"2021-02-11T06:12:06.134293+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T06:12:05.104971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T06:12:05.104971+0000","last_clean_scrub_stamp":"2021-02-11T06:12:05.104971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2021-02-11T06:12:06.133992+0000","last_change":"2021-02-11T06:12:06.129472+0000","last_active":"2021-02-11T06:12:06.133992+0000","last_peered":"2021-02-11T06:12:06.133992+0000","last_clean":"2021-02-11T06:12:06.133992+0000","last_became_active":"2021-02-11T06:12:06.129032+0000","last_became_peered":"2021-02-11T06:12:06.129032+0000","last_unstale":"2021-02-11T06:12:06.133992+0000","last_undegraded":"2021-02-11T06:12:06.133992+0000","last_fullsized":"2021-02-11T06:12:06.133992+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T06:12:05.104971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T06:12:05.104971+0000","last_clean_scrub_stamp":"2021-02-11T06:12:05.104971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2021-02-11T06:12:06.139442+0000","last_change":"2021-02-11T06:12:06.135316+0000","last_active":"2021-02-11T06:12:06.139442+0000","last_peered":"2021-02-11T06:12:06.139442+0000","last_clean":"2021-02-11T06:12:06.139442+0000","last_became_active":"2021-02-11T06:12:06.134933+0000","last_became_peered":"2021-02-11T06:12:06.134933+0000","last_unstale":"2021-02-11T06:12:06.139442+0000","last_undegraded":"2021-02-11T06:12:06.139442+0000","last_fullsized":"2021-02-11T06:12:06.139442+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T06:12:05.104971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T06:12:05.104971+0000","last_clean_scrub_stamp":"2021-02-11T06:12:05.104971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"12","reported_epoch":"14","state":"active+clean","last_fresh":"2021-02-11T06:12:06.115682+0000","last_change":"2021-02-11T06:11:58.970814+0000","last_active":"2021-02-11T06:12:06.115682+0000","last_peered":"2021-02-11T06:12:06.115682+0000","last_clean":"2021-02-11T06:12:06.115682+0000","last_became_active":"2021-02-11T06:11:58.969724+0000","last_became_peered":"2021-02-11T06:11:58.969724+0000","last_unstale":"2021-02-11T06:12:06.115682+0000","last_undegraded":"2021-02-11T06:12:06.115682+0000","last_fullsized":"2021-02-11T06:12:06.115682+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T06:11:57.925734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T06:11:57.925734+0000","last_clean_scrub_stamp":"2021-02-11T06:11:57.925734+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2021-02-11T06:12:06.136726+0000","last_change":"2021-02-11T06:12:06.132507+0000","last_active":"2021-02-11T06:12:06.136726+0000","last_peered":"2021-02-11T06:12:06.136726+0000","last_clean":"2021-02-11T06:12:06.136726+0000","last_became_active":"2021-02-11T06:12:06.132084+0000","last_became_peered":"2021-02-11T06:12:06.132084+0000","last_unstale":"2021-02-11T06:12:06.136726+0000","last_undegraded":"2021-02-11T06:12:06.136726+0000","last_fullsized":"2021-02-11T06:12:06.136726+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T06:12:05.104971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T06:12:05.104971+0000","last_clean_scrub_stamp":"2021-02-11T06:12:05.104971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2021-02-11T06:12:06.139428+0000","last_change":"2021-02-11T06:12:06.135133+0000","last_active":"2021-02-11T06:12:06.139428+0000","last_peered":"2021-02-11T06:12:06.139428+0000","last_clean":"2021-02-11T06:12:06.139428+0000","last_became_active":"2021-02-11T06:12:06.134730+0000","last_became_peered":"2021-02-11T06:12:06.134730+0000","last_unstale":"2021-02-11T06:12:06.139428+0000","last_undegraded":"2021-02-11T06:12:06.139428+0000","last_fullsized":"2021-02-11T06:12:06.139428+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T06:12:05.104971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T06:12:05.104971+0000","last_clean_scrub_stamp":"2021-02-11T06:12:05.104971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2021-02-11T06:12:06.136771+0000","last_change":"2021-02-11T06:12:06.132913+0000","last_active":"2021-02-11T06:12:06.136771+0000","last_peered":"2021-02-11T06:12:06.136771+0000","last_clean":"2021-02-11T06:12:06.136771+0000","last_became_active":"2021-02-11T06:12:06.132478+0000","last_became_peered":"2021-02-11T06:12:06.132478+0000","last_unstale":"2021-02-11T06:12:06.136771+0000","last_undegraded":"2021-02-11T06:12:06.136771+0000","last_fullsized":"2021-02-11T06:12:06.136771+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T06:12:05.104971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T06:12:05.104971+0000","last_clean_scrub_stamp":"2021-02-11T06:12:05.104971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2021-02-11T06:12:06.134844+0000","last_change":"2021-02-11T06:12:06.129927+0000","last_active":"2021-02-11T06:12:06.134844+0000","last_peered":"2021-02-11T06:12:06.134844+0000","last_clean":"2021-02-11T06:12:06.134844+0000","last_became_active":"2021-02-11T06:12:06.129535+0000","last_became_peered":"2021-02-11T06:12:06.129535+0000","last_unstale":"2021-02-11T06:12:06.134844+0000","last_undegraded":"2021-02-11T06:12:06.134844+0000","last_fullsized":"2021-02-11T06:12:06.134844+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T06:12:05.104971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T06:12:05.104971+0000","last_clean_scrub_stamp":"2021-02-11T06:12:05.104971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705671,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4936,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852664,"statfs":{"total":107374182400,"available":107369127936,"internally_reserved":0,"allocated":131072,"data_stored":37515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.446}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3680000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4936,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852664,"statfs":{"total":107374182400,"available":107369127936,"internally_reserved":0,"allocated":131072,"data_stored":37515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7769999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.171}]}]},{"osd":0,"up_from":9,"seq":38654705671,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4936,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852664,"statfs":{"total":107374182400,"available":107369127936,"internally_reserved":0,"allocated":131072,"data_stored":37515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3180000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4019999999999999}]}]}]}} 2021-02-11T06:12:16.376 INFO:tasks.ceph.ceph_manager.c2:clean! 2021-02-11T06:12:16.376 INFO:tasks.ceph:Waiting until ceph cluster c2 is healthy... 2021-02-11T06:12:16.376 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy 2021-02-11T06:12:16.376 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 health --format=json 2021-02-11T06:12:16.721 INFO:teuthology.orchestra.run.smithi172.stdout: 2021-02-11T06:12:16.735 INFO:teuthology.orchestra.run.smithi172.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2021-02-11T06:12:16.735 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy done 2021-02-11T06:12:16.735 INFO:teuthology.run_tasks:Running task rgw... 2021-02-11T06:12:16.766 DEBUG:tasks.rgw:config is {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}} 2021-02-11T06:12:16.766 DEBUG:tasks.rgw:client list is dict_keys(['c1.client.0', 'c1.client.1', 'c2.client.0', 'c2.client.1']) 2021-02-11T06:12:16.767 INFO:tasks.rgw:Creating data pools 2021-02-11T06:12:16.767 DEBUG:tasks.rgw:Obtaining remote for client c1.client.0 2021-02-11T06:12:16.767 DEBUG:teuthology.orchestra.run.smithi061:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2021-02-11T06:12:17.944 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'default.rgw.buckets.data' created 2021-02-11T06:12:33.563 DEBUG:teuthology.orchestra.run.smithi061:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2021-02-11T06:12:34.056 INFO:teuthology.orchestra.run.smithi061.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2021-02-11T06:12:34.071 DEBUG:teuthology.orchestra.run.smithi061:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2021-02-11T06:12:35.069 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'default.rgw.buckets.index' created 2021-02-11T06:12:35.095 DEBUG:teuthology.orchestra.run.smithi061:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2021-02-11T06:12:36.093 INFO:teuthology.orchestra.run.smithi061.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2021-02-11T06:12:36.114 DEBUG:tasks.rgw:Obtaining remote for client c1.client.1 2021-02-11T06:12:36.114 DEBUG:teuthology.orchestra.run.smithi061:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2021-02-11T06:12:36.411 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'default.rgw.buckets.data' already exists 2021-02-11T06:12:36.422 DEBUG:teuthology.orchestra.run.smithi061:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2021-02-11T06:12:37.078 INFO:teuthology.orchestra.run.smithi061.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2021-02-11T06:12:37.091 DEBUG:teuthology.orchestra.run.smithi061:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2021-02-11T06:12:37.385 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'default.rgw.buckets.index' already exists 2021-02-11T06:12:38.199 DEBUG:teuthology.orchestra.run.smithi061:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2021-02-11T06:12:39.098 INFO:teuthology.orchestra.run.smithi061.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2021-02-11T06:12:39.112 DEBUG:tasks.rgw:Obtaining remote for client c2.client.0 2021-02-11T06:12:39.112 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2021-02-11T06:12:40.154 INFO:teuthology.orchestra.run.smithi172.stderr:pool 'default.rgw.buckets.data' created 2021-02-11T06:12:40.167 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2021-02-11T06:12:41.178 INFO:teuthology.orchestra.run.smithi172.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2021-02-11T06:12:41.198 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2021-02-11T06:12:42.159 INFO:teuthology.orchestra.run.smithi172.stderr:pool 'default.rgw.buckets.index' created 2021-02-11T06:12:42.182 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2021-02-11T06:12:43.186 INFO:teuthology.orchestra.run.smithi172.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2021-02-11T06:12:45.375 DEBUG:tasks.rgw:Obtaining remote for client c2.client.1 2021-02-11T06:12:45.375 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2021-02-11T06:12:45.686 INFO:teuthology.orchestra.run.smithi172.stderr:pool 'default.rgw.buckets.data' already exists 2021-02-11T06:12:45.697 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2021-02-11T06:12:46.168 INFO:teuthology.orchestra.run.smithi172.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2021-02-11T06:12:46.183 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2021-02-11T06:12:46.475 INFO:teuthology.orchestra.run.smithi172.stderr:pool 'default.rgw.buckets.index' already exists 2021-02-11T06:13:04.360 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2021-02-11T06:13:05.164 INFO:teuthology.orchestra.run.smithi172.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2021-02-11T06:13:05.820 DEBUG:tasks.rgw:Pools created 2021-02-11T06:13:05.821 INFO:tasks.rgw:Configuring compression type = random 2021-02-11T06:13:05.821 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'list'] 2021-02-11T06:13:05.821 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'list'] 2021-02-11T06:13:05.821 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user list 2021-02-11T06:13:05.857 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:13:05.857 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:13:05.868 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:05.869+0000 7f4b1f6ad080 20 rados->read ofs=0 len=0 2021-02-11T06:13:07.079 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:07.077+0000 7f4b1f6ad080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T06:13:07.265 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:07.077+0000 7f4b1f6ad080 20 rados->read ofs=0 len=0 2021-02-11T06:13:07.271 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:07.081+0000 7f4b1f6ad080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T06:13:07.271 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:07.081+0000 7f4b1f6ad080 20 rados->read ofs=0 len=0 2021-02-11T06:13:07.272 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:07.081+0000 7f4b1f6ad080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T06:13:07.272 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:07.081+0000 7f4b1f6ad080 20 rados->read ofs=0 len=0 2021-02-11T06:13:07.272 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:07.085+0000 7f4b1f6ad080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T06:13:07.272 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:07.105+0000 7f4b1f6ad080 20 rados->read ofs=0 len=0 2021-02-11T06:13:07.272 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:07.105+0000 7f4b1f6ad080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T06:13:07.272 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:07.105+0000 7f4b1f6ad080 20 rados->read ofs=0 len=0 2021-02-11T06:13:07.272 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:07.105+0000 7f4b1f6ad080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T06:13:07.272 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:07.105+0000 7f4b1f6ad080 20 rados->read ofs=0 len=0 2021-02-11T06:13:07.273 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:07.105+0000 7f4b1f6ad080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T06:13:07.273 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:07.105+0000 7f4b1f6ad080 10 cannot find current period zonegroup using local zonegroup 2021-02-11T06:13:07.273 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:07.105+0000 7f4b1f6ad080 20 rados->read ofs=0 len=0 2021-02-11T06:13:07.273 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:07.105+0000 7f4b1f6ad080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T06:13:07.273 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:07.105+0000 7f4b1f6ad080 20 rados->read ofs=0 len=0 2021-02-11T06:13:07.273 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:07.105+0000 7f4b1f6ad080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T06:13:07.273 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:07.105+0000 7f4b1f6ad080 10 Creating default zonegroup 2021-02-11T06:13:07.273 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:07.105+0000 7f4b1f6ad080 20 get_system_obj_state: rctx=0x7ffd8cffa030 obj=:.pools.avail state=0x5600d65491f0 s->prefetch_data=0 2021-02-11T06:13:07.274 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:07.109+0000 7f4b1f6ad080 10 couldn't find old data placement pools config, setting up new ones for the zone 2021-02-11T06:13:07.274 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:07.121+0000 7f4b1f6ad080 10 WARNING: store->list_zones() returned r=-2 2021-02-11T06:13:07.274 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:07.121+0000 7f4b1f6ad080 20 rados->read ofs=0 len=0 2021-02-11T06:13:07.274 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:07.121+0000 7f4b1f6ad080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T06:13:07.274 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:07.129+0000 7f4b1f6ad080 20 rados->read ofs=0 len=0 2021-02-11T06:13:07.274 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:07.129+0000 7f4b1f6ad080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T06:13:07.274 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:07.129+0000 7f4b1f6ad080 10 could not read realm id: (2) No such file or directory 2021-02-11T06:13:07.274 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:07.129+0000 7f4b1f6ad080 10 WARNING: failed to set zone as default, r=-22 2021-02-11T06:13:07.275 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:07.129+0000 7f4b1f6ad080 20 rados->read ofs=0 len=0 2021-02-11T06:13:07.275 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:07.129+0000 7f4b1f6ad080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T06:13:07.275 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:07.133+0000 7f4b1f6ad080 20 rados->read ofs=0 len=0 2021-02-11T06:13:07.275 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:07.137+0000 7f4b1f6ad080 20 rados_obj.operate() r=0 bl.length=873 2021-02-11T06:13:07.275 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:07.137+0000 7f4b1f6ad080 20 rados->read ofs=0 len=0 2021-02-11T06:13:07.275 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:07.137+0000 7f4b1f6ad080 20 rados_obj.operate() r=0 bl.length=358 2021-02-11T06:13:07.275 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:07.137+0000 7f4b1f6ad080 20 zonegroup default 2021-02-11T06:13:07.276 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:07.137+0000 7f4b1f6ad080 20 rados->read ofs=0 len=0 2021-02-11T06:13:07.276 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:07.137+0000 7f4b1f6ad080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T06:13:07.276 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:07.137+0000 7f4b1f6ad080 10 Cannot find current period zone using local zone 2021-02-11T06:13:07.276 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:07.137+0000 7f4b1f6ad080 10 Using default name default 2021-02-11T06:13:07.276 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:07.137+0000 7f4b1f6ad080 20 rados->read ofs=0 len=0 2021-02-11T06:13:07.276 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:07.137+0000 7f4b1f6ad080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T06:13:07.276 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:07.137+0000 7f4b1f6ad080 20 rados->read ofs=0 len=0 2021-02-11T06:13:07.276 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:07.137+0000 7f4b1f6ad080 20 rados_obj.operate() r=0 bl.length=873 2021-02-11T06:13:07.276 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:07.137+0000 7f4b1f6ad080 20 zone default found 2021-02-11T06:13:07.276 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:07.137+0000 7f4b1f6ad080 20 rados->read ofs=0 len=0 2021-02-11T06:13:09.101 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:09.101+0000 7f4b1f6ad080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T06:13:09.102 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:09.101+0000 7f4b1f6ad080 20 rados->read ofs=0 len=0 2021-02-11T06:13:09.102 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:09.101+0000 7f4b1f6ad080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T06:13:09.102 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:09.101+0000 7f4b1f6ad080 20 started sync module instance, tier type = 2021-02-11T06:13:09.102 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:09.101+0000 7f4b1f6ad080 20 started zone id=4e8d7132-8fba-4855-8538-3deccc82e81c (name=default) with tier type = 2021-02-11T06:13:11.163 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:11.161+0000 7f4b1f6ad080 20 add_watcher() i=0 2021-02-11T06:13:11.163 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:11.161+0000 7f4b1f6ad080 20 add_watcher() i=1 2021-02-11T06:13:11.163 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:11.161+0000 7f4b1f6ad080 20 add_watcher() i=2 2021-02-11T06:13:11.163 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:11.161+0000 7f4b1f6ad080 20 add_watcher() i=3 2021-02-11T06:13:11.163 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:11.161+0000 7f4b1f6ad080 20 add_watcher() i=4 2021-02-11T06:13:11.163 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:11.161+0000 7f4b1f6ad080 20 add_watcher() i=5 2021-02-11T06:13:11.163 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:11.161+0000 7f4b1f6ad080 20 add_watcher() i=6 2021-02-11T06:13:11.165 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:11.165+0000 7f4b1f6ad080 20 add_watcher() i=7 2021-02-11T06:13:11.165 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:11.165+0000 7f4b1f6ad080 2 all 8 watchers are set, enabling cache 2021-02-11T06:13:11.193 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:11.193+0000 7f4a57fff700 2 RGWDataChangesLog::ChangesRenewThread: start 2021-02-11T06:13:11.193 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:11.193+0000 7f4b1f6ad080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-02-11T06:13:11.194 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:11.193+0000 7f4b1f6ad080 20 check_secure_mon_conn(): mode 1 is insecure 2021-02-11T06:13:15.128 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:15.129+0000 7f4b1f6ad080 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2021-02-11T06:13:25.490 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:15.261+0000 7f4a42ffd700 20 reqs_thread_entry: start 2021-02-11T06:13:25.490 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:15.261+0000 7f4b1f6ad080 20 init_complete bucket index max shards: 11 2021-02-11T06:13:25.490 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:15.261+0000 7f4a40ff9700 20 reqs_thread_entry: start 2021-02-11T06:13:25.490 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:15.261+0000 7f4b1f6ad080 10 rgw notify: Started notification manager with: 1 workers 2021-02-11T06:13:25.490 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:15.261+0000 7f4a32ffd700 20 rgw notify: INFO: next queues processing will happen at: Thu Feb 11 06:13:45 2021 2021-02-11T06:13:25.490 INFO:teuthology.orchestra.run.smithi061.stderr: 2021-02-11T06:13:25.490 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:15.261+0000 7f4a327fc700 20 reqs_thread_entry: start 2021-02-11T06:13:25.490 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:15.289+0000 7f4b1f6ad080 20 remove_watcher() i=0 2021-02-11T06:13:25.491 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:15.289+0000 7f4b1f6ad080 2 removed watcher, disabling cache 2021-02-11T06:13:25.491 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:15.293+0000 7f4b1f6ad080 20 remove_watcher() i=1 2021-02-11T06:13:25.491 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:15.293+0000 7f4b1f6ad080 20 remove_watcher() i=2 2021-02-11T06:13:25.491 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:15.297+0000 7f4b1f6ad080 20 remove_watcher() i=3 2021-02-11T06:13:25.491 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:15.301+0000 7f4b1f6ad080 20 remove_watcher() i=4 2021-02-11T06:13:25.491 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:15.301+0000 7f4b1f6ad080 20 remove_watcher() i=5 2021-02-11T06:13:25.491 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:15.305+0000 7f4b1f6ad080 20 remove_watcher() i=6 2021-02-11T06:13:25.491 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:15.309+0000 7f4b1f6ad080 20 remove_watcher() i=7 2021-02-11T06:13:25.491 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:15.309+0000 7f4a57fff700 2 RGWDataChangesLog::ChangesRenewThread: start 2021-02-11T06:13:25.491 INFO:teuthology.orchestra.run.smithi061.stdout:[] 2021-02-11T06:13:25.491 DEBUG:tasks.util.rgw: json result: [] 2021-02-11T06:13:25.492 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-02-11T06:13:25.492 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-02-11T06:13:25.492 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2021-02-11T06:13:25.572 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:13:25.573 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:13:25.587 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.585+0000 7f1c3703a080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-02-11T06:13:25.587 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.585+0000 7f1c3703a080 20 check_secure_mon_conn(): mode 1 is insecure 2021-02-11T06:13:25.594 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.593+0000 7f1b6bfff700 20 reqs_thread_entry: start 2021-02-11T06:13:25.594 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.593+0000 7f1c3703a080 20 rados->read ofs=0 len=0 2021-02-11T06:13:25.596 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.597+0000 7f1c3703a080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T06:13:25.596 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.597+0000 7f1c3703a080 20 rados->read ofs=0 len=0 2021-02-11T06:13:25.597 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.597+0000 7f1c3703a080 20 rados_obj.operate() r=0 bl.length=873 2021-02-11T06:13:25.597 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.597+0000 7f1c3703a080 20 rados->read ofs=0 len=0 2021-02-11T06:13:25.597 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.597+0000 7f1c3703a080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T06:13:25.597 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.597+0000 7f1c3703a080 20 rados->read ofs=0 len=0 2021-02-11T06:13:25.598 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.597+0000 7f1c3703a080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T06:13:25.598 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.597+0000 7f1c3703a080 20 rados->read ofs=0 len=0 2021-02-11T06:13:25.600 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.601+0000 7f1c3703a080 20 rados_obj.operate() r=0 bl.length=358 2021-02-11T06:13:25.610 INFO:teuthology.orchestra.run.smithi061.stdout:{"id":"4e8d7132-8fba-4855-8538-3deccc82e81c","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0}}],"realm_id":"","notif_pool":"default.rgw.log:notif"} 2021-02-11T06:13:25.610 DEBUG:tasks.util.rgw: json result: {'id': '4e8d7132-8fba-4855-8538-3deccc82e81c', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2021-02-11T06:13:25.610 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['user', 'list'] 2021-02-11T06:13:25.610 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'user', 'list'] 2021-02-11T06:13:25.610 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 user list 2021-02-11T06:13:25.691 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:13:25.691 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:13:25.704 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.705+0000 7f7f541d3080 20 rados->read ofs=0 len=0 2021-02-11T06:13:25.708 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.709+0000 7f7f541d3080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T06:13:25.708 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.709+0000 7f7f541d3080 20 rados->read ofs=0 len=0 2021-02-11T06:13:25.708 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.709+0000 7f7f541d3080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T06:13:25.708 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.709+0000 7f7f541d3080 20 rados->read ofs=0 len=0 2021-02-11T06:13:25.709 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.709+0000 7f7f541d3080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T06:13:25.709 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.709+0000 7f7f541d3080 20 rados->read ofs=0 len=0 2021-02-11T06:13:25.709 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.709+0000 7f7f541d3080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T06:13:25.738 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.737+0000 7f7f541d3080 20 RGWRados::pool_iterate: got zone_info.4e8d7132-8fba-4855-8538-3deccc82e81c 2021-02-11T06:13:25.738 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.737+0000 7f7f541d3080 20 RGWRados::pool_iterate: got zonegroup_info.9d1d703e-6a0a-4888-91fd-494c79c73c85 2021-02-11T06:13:25.738 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.737+0000 7f7f541d3080 20 RGWRados::pool_iterate: got zone_names.default 2021-02-11T06:13:25.738 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.737+0000 7f7f541d3080 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-02-11T06:13:25.738 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.737+0000 7f7f541d3080 20 rados->read ofs=0 len=0 2021-02-11T06:13:25.739 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.737+0000 7f7f541d3080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T06:13:25.739 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.737+0000 7f7f541d3080 20 rados->read ofs=0 len=0 2021-02-11T06:13:25.740 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.741+0000 7f7f541d3080 20 rados_obj.operate() r=0 bl.length=889 2021-02-11T06:13:25.741 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.741+0000 7f7f541d3080 20 rados->read ofs=0 len=0 2021-02-11T06:13:25.741 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.741+0000 7f7f541d3080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T06:13:25.741 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.741+0000 7f7f541d3080 20 rados->read ofs=0 len=0 2021-02-11T06:13:25.742 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.741+0000 7f7f541d3080 20 rados_obj.operate() r=0 bl.length=358 2021-02-11T06:13:25.742 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.741+0000 7f7f541d3080 20 rados->read ofs=0 len=0 2021-02-11T06:13:25.743 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.741+0000 7f7f541d3080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T06:13:25.743 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.741+0000 7f7f541d3080 10 cannot find current period zonegroup using local zonegroup 2021-02-11T06:13:25.743 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.741+0000 7f7f541d3080 20 rados->read ofs=0 len=0 2021-02-11T06:13:25.744 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.745+0000 7f7f541d3080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T06:13:25.744 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.745+0000 7f7f541d3080 20 rados->read ofs=0 len=0 2021-02-11T06:13:25.745 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.745+0000 7f7f541d3080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T06:13:25.745 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.745+0000 7f7f541d3080 20 rados->read ofs=0 len=0 2021-02-11T06:13:25.745 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.745+0000 7f7f541d3080 20 rados_obj.operate() r=0 bl.length=358 2021-02-11T06:13:25.745 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.745+0000 7f7f541d3080 20 zonegroup default 2021-02-11T06:13:25.746 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.745+0000 7f7f541d3080 20 rados->read ofs=0 len=0 2021-02-11T06:13:25.746 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.745+0000 7f7f541d3080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T06:13:25.746 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.745+0000 7f7f541d3080 10 Cannot find current period zone using local zone 2021-02-11T06:13:25.746 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.745+0000 7f7f541d3080 20 rados->read ofs=0 len=0 2021-02-11T06:13:25.747 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.749+0000 7f7f541d3080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T06:13:25.747 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.749+0000 7f7f541d3080 20 rados->read ofs=0 len=0 2021-02-11T06:13:25.748 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.749+0000 7f7f541d3080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T06:13:25.748 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.749+0000 7f7f541d3080 20 rados->read ofs=0 len=0 2021-02-11T06:13:25.749 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.749+0000 7f7f541d3080 20 rados_obj.operate() r=0 bl.length=889 2021-02-11T06:13:25.749 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.749+0000 7f7f541d3080 20 zone default found 2021-02-11T06:13:25.749 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.749+0000 7f7f541d3080 20 rados->read ofs=0 len=0 2021-02-11T06:13:25.749 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.749+0000 7f7f541d3080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T06:13:25.749 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.749+0000 7f7f541d3080 20 rados->read ofs=0 len=0 2021-02-11T06:13:25.751 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.749+0000 7f7f541d3080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T06:13:25.751 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.749+0000 7f7f541d3080 20 started sync module instance, tier type = 2021-02-11T06:13:25.751 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.749+0000 7f7f541d3080 20 started zone id=4e8d7132-8fba-4855-8538-3deccc82e81c (name=default) with tier type = 2021-02-11T06:13:25.775 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.777+0000 7f7f541d3080 20 add_watcher() i=0 2021-02-11T06:13:25.775 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.777+0000 7f7f541d3080 20 add_watcher() i=1 2021-02-11T06:13:25.775 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.777+0000 7f7f541d3080 20 add_watcher() i=2 2021-02-11T06:13:25.776 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.777+0000 7f7f541d3080 20 add_watcher() i=3 2021-02-11T06:13:25.776 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.777+0000 7f7f541d3080 20 add_watcher() i=4 2021-02-11T06:13:25.776 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.777+0000 7f7f541d3080 20 add_watcher() i=5 2021-02-11T06:13:25.776 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.777+0000 7f7f541d3080 20 add_watcher() i=6 2021-02-11T06:13:25.777 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.777+0000 7f7f541d3080 20 add_watcher() i=7 2021-02-11T06:13:25.777 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.777+0000 7f7f541d3080 2 all 8 watchers are set, enabling cache 2021-02-11T06:13:25.798 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.797+0000 7f7e8bfff700 2 RGWDataChangesLog::ChangesRenewThread: start 2021-02-11T06:13:25.798 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.797+0000 7f7f541d3080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-02-11T06:13:25.798 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.797+0000 7f7f541d3080 20 check_secure_mon_conn(): mode 1 is insecure 2021-02-11T06:13:25.898 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.897+0000 7f7e7affd700 20 reqs_thread_entry: start 2021-02-11T06:13:37.005 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.897+0000 7f7e78ff9700 20 reqs_thread_entry: start 2021-02-11T06:13:37.006 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.897+0000 7f7f541d3080 20 init_complete bucket index max shards: 11 2021-02-11T06:13:37.006 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.897+0000 7f7f541d3080 10 rgw notify: Started notification manager with: 1 workers 2021-02-11T06:13:37.006 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.897+0000 7f7e72ffd700 20 rgw notify: INFO: next queues processing will happen at: Thu Feb 11 06:13:56 2021 2021-02-11T06:13:37.006 INFO:teuthology.orchestra.run.smithi061.stderr: 2021-02-11T06:13:37.006 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.897+0000 7f7e727fc700 20 reqs_thread_entry: start 2021-02-11T06:13:37.006 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.929+0000 7f7f541d3080 20 remove_watcher() i=0 2021-02-11T06:13:37.006 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.929+0000 7f7f541d3080 2 removed watcher, disabling cache 2021-02-11T06:13:37.006 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.933+0000 7f7f541d3080 20 remove_watcher() i=1 2021-02-11T06:13:37.006 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.937+0000 7f7f541d3080 20 remove_watcher() i=2 2021-02-11T06:13:37.006 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.941+0000 7f7f541d3080 20 remove_watcher() i=3 2021-02-11T06:13:37.006 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.941+0000 7f7f541d3080 20 remove_watcher() i=4 2021-02-11T06:13:37.006 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.945+0000 7f7f541d3080 20 remove_watcher() i=5 2021-02-11T06:13:37.006 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.949+0000 7f7f541d3080 20 remove_watcher() i=6 2021-02-11T06:13:37.007 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.949+0000 7f7f541d3080 20 remove_watcher() i=7 2021-02-11T06:13:37.007 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:25.953+0000 7f7e8bfff700 2 RGWDataChangesLog::ChangesRenewThread: start 2021-02-11T06:13:37.007 INFO:teuthology.orchestra.run.smithi061.stdout:[] 2021-02-11T06:13:37.007 DEBUG:tasks.util.rgw: json result: [] 2021-02-11T06:13:37.007 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-02-11T06:13:37.007 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-02-11T06:13:37.007 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2021-02-11T06:13:37.086 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:13:37.086 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:13:37.098 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:37.097+0000 7f89bedf0080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-02-11T06:13:37.098 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:37.097+0000 7f89bedf0080 20 check_secure_mon_conn(): mode 1 is insecure 2021-02-11T06:13:37.105 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:37.105+0000 7f88f3fff700 20 reqs_thread_entry: start 2021-02-11T06:13:37.105 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:37.105+0000 7f89bedf0080 20 rados->read ofs=0 len=0 2021-02-11T06:13:37.107 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:37.105+0000 7f89bedf0080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T06:13:37.107 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:37.105+0000 7f89bedf0080 20 rados->read ofs=0 len=0 2021-02-11T06:13:37.107 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:37.109+0000 7f89bedf0080 20 rados_obj.operate() r=0 bl.length=889 2021-02-11T06:13:37.107 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:37.109+0000 7f89bedf0080 20 rados->read ofs=0 len=0 2021-02-11T06:13:37.108 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:37.109+0000 7f89bedf0080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T06:13:37.108 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:37.109+0000 7f89bedf0080 20 rados->read ofs=0 len=0 2021-02-11T06:13:37.108 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:37.109+0000 7f89bedf0080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T06:13:37.108 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:37.109+0000 7f89bedf0080 20 rados->read ofs=0 len=0 2021-02-11T06:13:37.109 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:37.109+0000 7f89bedf0080 20 rados_obj.operate() r=0 bl.length=358 2021-02-11T06:13:37.118 INFO:teuthology.orchestra.run.smithi061.stdout:{"id":"4e8d7132-8fba-4855-8538-3deccc82e81c","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0}}],"realm_id":"","notif_pool":"default.rgw.log:notif"} 2021-02-11T06:13:37.119 DEBUG:tasks.util.rgw: json result: {'id': '4e8d7132-8fba-4855-8538-3deccc82e81c', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2021-02-11T06:13:37.119 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['user', 'list'] 2021-02-11T06:13:37.119 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'user', 'list'] 2021-02-11T06:13:37.119 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 user list 2021-02-11T06:13:37.154 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:13:37.154 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:13:37.165 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:37.168+0000 7f9d1d400080 20 rados->read ofs=0 len=0 2021-02-11T06:13:39.191 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:39.192+0000 7f9d1d400080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T06:13:39.192 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:39.192+0000 7f9d1d400080 20 rados->read ofs=0 len=0 2021-02-11T06:13:39.200 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:39.200+0000 7f9d1d400080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T06:13:39.201 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:39.200+0000 7f9d1d400080 20 rados->read ofs=0 len=0 2021-02-11T06:13:39.201 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:39.200+0000 7f9d1d400080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T06:13:39.201 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:39.200+0000 7f9d1d400080 20 rados->read ofs=0 len=0 2021-02-11T06:13:39.201 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:39.204+0000 7f9d1d400080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T06:13:39.219 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:39.220+0000 7f9d1d400080 20 rados->read ofs=0 len=0 2021-02-11T06:13:39.220 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:39.220+0000 7f9d1d400080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T06:13:39.220 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:39.220+0000 7f9d1d400080 20 rados->read ofs=0 len=0 2021-02-11T06:13:39.220 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:39.220+0000 7f9d1d400080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T06:13:39.220 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:39.220+0000 7f9d1d400080 20 rados->read ofs=0 len=0 2021-02-11T06:13:39.220 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:39.220+0000 7f9d1d400080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T06:13:39.220 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:39.220+0000 7f9d1d400080 10 cannot find current period zonegroup using local zonegroup 2021-02-11T06:13:39.220 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:39.220+0000 7f9d1d400080 20 rados->read ofs=0 len=0 2021-02-11T06:13:39.221 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:39.220+0000 7f9d1d400080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T06:13:39.221 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:39.220+0000 7f9d1d400080 20 rados->read ofs=0 len=0 2021-02-11T06:13:39.221 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:39.220+0000 7f9d1d400080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T06:13:39.221 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:39.220+0000 7f9d1d400080 10 Creating default zonegroup 2021-02-11T06:13:39.221 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:39.220+0000 7f9d1d400080 20 get_system_obj_state: rctx=0x7fffc0a79780 obj=:.pools.avail state=0x555df0f411f0 s->prefetch_data=0 2021-02-11T06:13:39.221 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:39.220+0000 7f9d1d400080 10 couldn't find old data placement pools config, setting up new ones for the zone 2021-02-11T06:13:39.236 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:39.236+0000 7f9d1d400080 10 WARNING: store->list_zones() returned r=-2 2021-02-11T06:13:39.237 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:39.236+0000 7f9d1d400080 20 rados->read ofs=0 len=0 2021-02-11T06:13:39.237 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:39.236+0000 7f9d1d400080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T06:13:39.245 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:39.244+0000 7f9d1d400080 20 rados->read ofs=0 len=0 2021-02-11T06:13:39.245 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:39.248+0000 7f9d1d400080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T06:13:39.245 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:39.248+0000 7f9d1d400080 10 could not read realm id: (2) No such file or directory 2021-02-11T06:13:39.246 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:39.248+0000 7f9d1d400080 10 WARNING: failed to set zone as default, r=-22 2021-02-11T06:13:39.246 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:39.248+0000 7f9d1d400080 20 rados->read ofs=0 len=0 2021-02-11T06:13:39.246 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:39.248+0000 7f9d1d400080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T06:13:39.252 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:39.252+0000 7f9d1d400080 20 rados->read ofs=0 len=0 2021-02-11T06:13:39.252 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:39.252+0000 7f9d1d400080 20 rados_obj.operate() r=0 bl.length=873 2021-02-11T06:13:39.252 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:39.252+0000 7f9d1d400080 20 rados->read ofs=0 len=0 2021-02-11T06:13:39.253 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:39.256+0000 7f9d1d400080 20 rados_obj.operate() r=0 bl.length=358 2021-02-11T06:13:39.253 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:39.256+0000 7f9d1d400080 20 zonegroup default 2021-02-11T06:13:39.253 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:39.256+0000 7f9d1d400080 20 rados->read ofs=0 len=0 2021-02-11T06:13:39.254 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:39.256+0000 7f9d1d400080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T06:13:39.254 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:39.256+0000 7f9d1d400080 10 Cannot find current period zone using local zone 2021-02-11T06:13:39.254 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:39.256+0000 7f9d1d400080 10 Using default name default 2021-02-11T06:13:39.254 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:39.256+0000 7f9d1d400080 20 rados->read ofs=0 len=0 2021-02-11T06:13:39.255 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:39.256+0000 7f9d1d400080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T06:13:39.255 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:39.256+0000 7f9d1d400080 20 rados->read ofs=0 len=0 2021-02-11T06:13:39.256 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:39.256+0000 7f9d1d400080 20 rados_obj.operate() r=0 bl.length=873 2021-02-11T06:13:39.256 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:39.256+0000 7f9d1d400080 20 zone default found 2021-02-11T06:13:39.256 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:39.256+0000 7f9d1d400080 20 rados->read ofs=0 len=0 2021-02-11T06:13:41.219 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:41.220+0000 7f9d1d400080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T06:13:41.219 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:41.220+0000 7f9d1d400080 20 rados->read ofs=0 len=0 2021-02-11T06:13:41.220 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:41.220+0000 7f9d1d400080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T06:13:41.220 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:41.220+0000 7f9d1d400080 20 started sync module instance, tier type = 2021-02-11T06:13:41.220 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:41.220+0000 7f9d1d400080 20 started zone id=9979490b-9dbd-4b47-8f20-76b2ddd377e6 (name=default) with tier type = 2021-02-11T06:13:43.266 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:43.268+0000 7f9d1d400080 20 add_watcher() i=0 2021-02-11T06:13:43.266 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:43.268+0000 7f9d1d400080 20 add_watcher() i=1 2021-02-11T06:13:43.266 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:43.268+0000 7f9d1d400080 20 add_watcher() i=2 2021-02-11T06:13:43.266 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:43.268+0000 7f9d1d400080 20 add_watcher() i=3 2021-02-11T06:13:43.267 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:43.268+0000 7f9d1d400080 20 add_watcher() i=4 2021-02-11T06:13:43.267 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:43.268+0000 7f9d1d400080 20 add_watcher() i=5 2021-02-11T06:13:43.267 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:43.268+0000 7f9d1d400080 20 add_watcher() i=6 2021-02-11T06:13:43.268 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:43.268+0000 7f9d1d400080 20 add_watcher() i=7 2021-02-11T06:13:43.268 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:43.268+0000 7f9d1d400080 2 all 8 watchers are set, enabling cache 2021-02-11T06:13:43.290 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:43.292+0000 7f9cda7e4700 2 RGWDataChangesLog::ChangesRenewThread: start 2021-02-11T06:13:43.290 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:43.292+0000 7f9d1d400080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-02-11T06:13:43.290 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:43.292+0000 7f9d1d400080 20 check_secure_mon_conn(): mode 1 is insecure 2021-02-11T06:13:47.218 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:47.220+0000 7f9d1d400080 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2021-02-11T06:13:52.741 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:47.340+0000 7f9c497fa700 20 reqs_thread_entry: start 2021-02-11T06:13:52.741 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:47.340+0000 7f9c2effd700 20 reqs_thread_entry: start 2021-02-11T06:13:52.741 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:47.340+0000 7f9d1d400080 20 init_complete bucket index max shards: 11 2021-02-11T06:13:52.741 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:47.340+0000 7f9d1d400080 10 rgw notify: Started notification manager with: 1 workers 2021-02-11T06:13:52.741 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:47.340+0000 7f9c2cff9700 20 reqs_thread_entry: start 2021-02-11T06:13:52.741 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:47.340+0000 7f9c2d7fa700 20 rgw notify: INFO: next queues processing will happen at: Thu Feb 11 06:14:17 2021 2021-02-11T06:13:52.741 INFO:teuthology.orchestra.run.smithi172.stderr: 2021-02-11T06:13:52.741 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:47.368+0000 7f9d1d400080 20 remove_watcher() i=0 2021-02-11T06:13:52.742 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:47.368+0000 7f9d1d400080 2 removed watcher, disabling cache 2021-02-11T06:13:52.742 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:47.368+0000 7f9d1d400080 20 remove_watcher() i=1 2021-02-11T06:13:52.742 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:47.372+0000 7f9d1d400080 20 remove_watcher() i=2 2021-02-11T06:13:52.742 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:47.376+0000 7f9d1d400080 20 remove_watcher() i=3 2021-02-11T06:13:52.742 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:47.376+0000 7f9d1d400080 20 remove_watcher() i=4 2021-02-11T06:13:52.742 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:47.380+0000 7f9d1d400080 20 remove_watcher() i=5 2021-02-11T06:13:52.742 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:47.384+0000 7f9d1d400080 20 remove_watcher() i=6 2021-02-11T06:13:52.742 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:47.384+0000 7f9d1d400080 20 remove_watcher() i=7 2021-02-11T06:13:52.742 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:47.388+0000 7f9cda7e4700 2 RGWDataChangesLog::ChangesRenewThread: start 2021-02-11T06:13:52.742 INFO:teuthology.orchestra.run.smithi172.stdout:[] 2021-02-11T06:13:52.743 DEBUG:tasks.util.rgw: json result: [] 2021-02-11T06:13:52.743 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-02-11T06:13:52.743 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-02-11T06:13:52.743 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2021-02-11T06:13:52.825 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:13:52.826 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:13:52.837 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:52.840+0000 7f168e2b3080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-02-11T06:13:52.838 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:52.840+0000 7f168e2b3080 20 check_secure_mon_conn(): mode 1 is insecure 2021-02-11T06:13:52.844 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:52.844+0000 7f168e2b3080 20 rados->read ofs=0 len=0 2021-02-11T06:13:52.844 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:52.844+0000 7f15c27fc700 20 reqs_thread_entry: start 2021-02-11T06:13:52.846 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:52.848+0000 7f168e2b3080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T06:13:52.846 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:52.848+0000 7f168e2b3080 20 rados->read ofs=0 len=0 2021-02-11T06:13:52.846 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:52.848+0000 7f168e2b3080 20 rados_obj.operate() r=0 bl.length=873 2021-02-11T06:13:52.846 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:52.848+0000 7f168e2b3080 20 rados->read ofs=0 len=0 2021-02-11T06:13:52.847 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:52.848+0000 7f168e2b3080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T06:13:52.847 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:52.848+0000 7f168e2b3080 20 rados->read ofs=0 len=0 2021-02-11T06:13:52.848 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:52.848+0000 7f168e2b3080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T06:13:52.848 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:52.848+0000 7f168e2b3080 20 rados->read ofs=0 len=0 2021-02-11T06:13:52.848 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:52.848+0000 7f168e2b3080 20 rados_obj.operate() r=0 bl.length=358 2021-02-11T06:13:52.858 INFO:teuthology.orchestra.run.smithi172.stdout:{"id":"9979490b-9dbd-4b47-8f20-76b2ddd377e6","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0}}],"realm_id":"","notif_pool":"default.rgw.log:notif"} 2021-02-11T06:13:52.858 DEBUG:tasks.util.rgw: json result: {'id': '9979490b-9dbd-4b47-8f20-76b2ddd377e6', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2021-02-11T06:13:52.858 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['user', 'list'] 2021-02-11T06:13:52.858 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'user', 'list'] 2021-02-11T06:13:52.859 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 user list 2021-02-11T06:13:52.936 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:13:52.936 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:13:52.947 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:52.948+0000 7fc8a65ec080 20 rados->read ofs=0 len=0 2021-02-11T06:13:52.950 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:52.952+0000 7fc8a65ec080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T06:13:52.950 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:52.952+0000 7fc8a65ec080 20 rados->read ofs=0 len=0 2021-02-11T06:13:52.950 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:52.952+0000 7fc8a65ec080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T06:13:52.951 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:52.952+0000 7fc8a65ec080 20 rados->read ofs=0 len=0 2021-02-11T06:13:52.951 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:52.952+0000 7fc8a65ec080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T06:13:52.951 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:52.952+0000 7fc8a65ec080 20 rados->read ofs=0 len=0 2021-02-11T06:13:52.952 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:52.952+0000 7fc8a65ec080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T06:13:52.973 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:52.976+0000 7fc8a65ec080 20 RGWRados::pool_iterate: got zonegroup_info.20876384-0013-4970-869b-33090e73d0bc 2021-02-11T06:13:52.973 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:52.976+0000 7fc8a65ec080 20 RGWRados::pool_iterate: got zone_info.9979490b-9dbd-4b47-8f20-76b2ddd377e6 2021-02-11T06:13:52.974 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:52.976+0000 7fc8a65ec080 20 RGWRados::pool_iterate: got zone_names.default 2021-02-11T06:13:52.974 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:52.976+0000 7fc8a65ec080 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-02-11T06:13:52.974 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:52.976+0000 7fc8a65ec080 20 rados->read ofs=0 len=0 2021-02-11T06:13:52.974 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:52.976+0000 7fc8a65ec080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T06:13:52.974 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:52.976+0000 7fc8a65ec080 20 rados->read ofs=0 len=0 2021-02-11T06:13:52.975 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:52.976+0000 7fc8a65ec080 20 rados_obj.operate() r=0 bl.length=889 2021-02-11T06:13:52.975 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:52.976+0000 7fc8a65ec080 20 rados->read ofs=0 len=0 2021-02-11T06:13:52.976 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:52.976+0000 7fc8a65ec080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T06:13:52.976 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:52.976+0000 7fc8a65ec080 20 rados->read ofs=0 len=0 2021-02-11T06:13:52.977 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:52.976+0000 7fc8a65ec080 20 rados_obj.operate() r=0 bl.length=358 2021-02-11T06:13:52.977 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:52.976+0000 7fc8a65ec080 20 rados->read ofs=0 len=0 2021-02-11T06:13:52.977 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:52.980+0000 7fc8a65ec080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T06:13:52.978 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:52.980+0000 7fc8a65ec080 10 cannot find current period zonegroup using local zonegroup 2021-02-11T06:13:52.978 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:52.980+0000 7fc8a65ec080 20 rados->read ofs=0 len=0 2021-02-11T06:13:52.978 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:52.980+0000 7fc8a65ec080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T06:13:52.978 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:52.980+0000 7fc8a65ec080 20 rados->read ofs=0 len=0 2021-02-11T06:13:52.979 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:52.980+0000 7fc8a65ec080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T06:13:52.979 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:52.980+0000 7fc8a65ec080 20 rados->read ofs=0 len=0 2021-02-11T06:13:52.980 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:52.980+0000 7fc8a65ec080 20 rados_obj.operate() r=0 bl.length=358 2021-02-11T06:13:52.980 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:52.980+0000 7fc8a65ec080 20 zonegroup default 2021-02-11T06:13:52.980 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:52.980+0000 7fc8a65ec080 20 rados->read ofs=0 len=0 2021-02-11T06:13:52.980 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:52.980+0000 7fc8a65ec080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T06:13:52.981 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:52.980+0000 7fc8a65ec080 10 Cannot find current period zone using local zone 2021-02-11T06:13:52.981 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:52.980+0000 7fc8a65ec080 20 rados->read ofs=0 len=0 2021-02-11T06:13:52.981 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:52.984+0000 7fc8a65ec080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T06:13:52.981 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:52.984+0000 7fc8a65ec080 20 rados->read ofs=0 len=0 2021-02-11T06:13:52.982 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:52.984+0000 7fc8a65ec080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T06:13:52.982 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:52.984+0000 7fc8a65ec080 20 rados->read ofs=0 len=0 2021-02-11T06:13:52.983 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:52.984+0000 7fc8a65ec080 20 rados_obj.operate() r=0 bl.length=889 2021-02-11T06:13:52.983 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:52.984+0000 7fc8a65ec080 20 zone default found 2021-02-11T06:13:52.983 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:52.984+0000 7fc8a65ec080 20 rados->read ofs=0 len=0 2021-02-11T06:13:52.983 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:52.984+0000 7fc8a65ec080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T06:13:52.984 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:52.984+0000 7fc8a65ec080 20 rados->read ofs=0 len=0 2021-02-11T06:13:52.984 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:52.984+0000 7fc8a65ec080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T06:13:52.984 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:52.984+0000 7fc8a65ec080 20 started sync module instance, tier type = 2021-02-11T06:13:52.984 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:52.984+0000 7fc8a65ec080 20 started zone id=9979490b-9dbd-4b47-8f20-76b2ddd377e6 (name=default) with tier type = 2021-02-11T06:13:53.015 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:53.016+0000 7fc8a65ec080 20 add_watcher() i=0 2021-02-11T06:13:53.015 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:53.016+0000 7fc8a65ec080 20 add_watcher() i=1 2021-02-11T06:13:53.015 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:53.016+0000 7fc8a65ec080 20 add_watcher() i=2 2021-02-11T06:13:53.015 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:53.016+0000 7fc8a65ec080 20 add_watcher() i=3 2021-02-11T06:13:53.015 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:53.016+0000 7fc8a65ec080 20 add_watcher() i=4 2021-02-11T06:13:53.015 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:53.016+0000 7fc8a65ec080 20 add_watcher() i=5 2021-02-11T06:13:53.016 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:53.016+0000 7fc8a65ec080 20 add_watcher() i=6 2021-02-11T06:13:53.017 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:53.020+0000 7fc8a65ec080 20 add_watcher() i=7 2021-02-11T06:13:53.018 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:53.020+0000 7fc8a65ec080 2 all 8 watchers are set, enabling cache 2021-02-11T06:13:53.046 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:53.048+0000 7fc863fdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2021-02-11T06:13:53.046 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:53.048+0000 7fc8a65ec080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-02-11T06:13:53.046 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:53.048+0000 7fc8a65ec080 20 check_secure_mon_conn(): mode 1 is insecure 2021-02-11T06:13:53.145 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:53.144+0000 7fc7d2ffd700 20 reqs_thread_entry: start 2021-02-11T06:13:53.145 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:53.144+0000 7fc7d0ff9700 20 reqs_thread_entry: start 2021-02-11T06:13:53.145 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:53.144+0000 7fc8a65ec080 20 init_complete bucket index max shards: 11 2021-02-11T06:13:53.145 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:53.148+0000 7fc8a65ec080 10 rgw notify: Started notification manager with: 1 workers 2021-02-11T06:13:53.146 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:53.148+0000 7fc7caffd700 20 rgw notify: INFO: next queues processing will happen at: Thu Feb 11 06:14:23 2021 2021-02-11T06:13:53.146 INFO:teuthology.orchestra.run.smithi172.stderr: 2021-02-11T06:13:53.146 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:53.148+0000 7fc7ca7fc700 20 reqs_thread_entry: start 2021-02-11T06:13:53.175 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:53.176+0000 7fc8a65ec080 20 remove_watcher() i=0 2021-02-11T06:13:53.175 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:53.176+0000 7fc8a65ec080 2 removed watcher, disabling cache 2021-02-11T06:13:53.177 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:53.180+0000 7fc8a65ec080 20 remove_watcher() i=1 2021-02-11T06:13:53.181 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:53.180+0000 7fc8a65ec080 20 remove_watcher() i=2 2021-02-11T06:13:53.183 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:53.184+0000 7fc8a65ec080 20 remove_watcher() i=3 2021-02-11T06:13:53.186 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:53.188+0000 7fc8a65ec080 20 remove_watcher() i=4 2021-02-11T06:13:53.189 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:53.188+0000 7fc8a65ec080 20 remove_watcher() i=5 2021-02-11T06:13:53.191 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:53.192+0000 7fc8a65ec080 20 remove_watcher() i=6 2021-02-11T06:13:53.194 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:53.196+0000 7fc8a65ec080 20 remove_watcher() i=7 2021-02-11T06:13:53.195 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:53.196+0000 7fc863fdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2021-02-11T06:13:53.202 INFO:teuthology.orchestra.run.smithi172.stdout:[] 2021-02-11T06:13:53.202 DEBUG:tasks.util.rgw: json result: [] 2021-02-11T06:13:53.202 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-02-11T06:13:53.202 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-02-11T06:13:53.203 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2021-02-11T06:13:53.286 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:13:53.286 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:13:53.297 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:53.296+0000 7f649e41f080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-02-11T06:13:53.297 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:53.296+0000 7f649e41f080 20 check_secure_mon_conn(): mode 1 is insecure 2021-02-11T06:13:53.302 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:53.304+0000 7f63d3fff700 20 reqs_thread_entry: start 2021-02-11T06:13:53.302 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:53.304+0000 7f649e41f080 20 rados->read ofs=0 len=0 2021-02-11T06:13:53.305 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:53.304+0000 7f649e41f080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T06:13:53.305 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:53.304+0000 7f649e41f080 20 rados->read ofs=0 len=0 2021-02-11T06:13:53.305 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:53.308+0000 7f649e41f080 20 rados_obj.operate() r=0 bl.length=889 2021-02-11T06:13:53.306 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:53.308+0000 7f649e41f080 20 rados->read ofs=0 len=0 2021-02-11T06:13:53.306 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:53.308+0000 7f649e41f080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T06:13:53.306 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:53.308+0000 7f649e41f080 20 rados->read ofs=0 len=0 2021-02-11T06:13:53.307 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:53.308+0000 7f649e41f080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T06:13:53.307 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:53.308+0000 7f649e41f080 20 rados->read ofs=0 len=0 2021-02-11T06:13:53.307 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:13:53.308+0000 7f649e41f080 20 rados_obj.operate() r=0 bl.length=358 2021-02-11T06:13:53.316 INFO:teuthology.orchestra.run.smithi172.stdout:{"id":"9979490b-9dbd-4b47-8f20-76b2ddd377e6","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0}}],"realm_id":"","notif_pool":"default.rgw.log:notif"} 2021-02-11T06:13:53.317 DEBUG:tasks.util.rgw: json result: {'id': '9979490b-9dbd-4b47-8f20-76b2ddd377e6', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2021-02-11T06:13:53.317 INFO:tasks.rgw:Starting rgw... 2021-02-11T06:13:53.318 INFO:tasks.rgw:rgw c1.client.0 config is {'port': 8000} 2021-02-11T06:13:53.318 INFO:tasks.rgw:Using beast as radosgw frontend 2021-02-11T06:13:53.318 INFO:tasks.rgw.c1.client.0:Restarting daemon 2021-02-11T06:13:53.318 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2021-02-11T06:13:53.321 INFO:tasks.rgw.c1.client.0:Started 2021-02-11T06:13:53.321 INFO:tasks.rgw:rgw c1.client.1 config is {'port': 8001} 2021-02-11T06:13:53.321 INFO:tasks.rgw:Using beast as radosgw frontend 2021-02-11T06:13:53.321 INFO:tasks.rgw.c1.client.1:Restarting daemon 2021-02-11T06:13:53.321 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2021-02-11T06:13:53.323 INFO:tasks.rgw.c1.client.1:Started 2021-02-11T06:13:53.323 INFO:tasks.rgw:rgw c2.client.0 config is {'port': 8000} 2021-02-11T06:13:53.323 INFO:tasks.rgw:Using beast as radosgw frontend 2021-02-11T06:13:53.324 INFO:tasks.rgw.c2.client.0:Restarting daemon 2021-02-11T06:13:53.324 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2021-02-11T06:13:53.358 INFO:tasks.rgw.c2.client.0:Started 2021-02-11T06:13:53.359 INFO:tasks.rgw:rgw c2.client.1 config is {'port': 8001} 2021-02-11T06:13:53.359 INFO:tasks.rgw:Using beast as radosgw frontend 2021-02-11T06:13:53.359 INFO:tasks.rgw.c2.client.1:Restarting daemon 2021-02-11T06:13:53.359 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2021-02-11T06:13:53.361 INFO:tasks.rgw.c2.client.1:Started 2021-02-11T06:13:53.361 INFO:tasks.rgw:Polling c1.client.0 until it starts accepting connections on http://smithi061.front.sepia.ceph.com:8000/ 2021-02-11T06:13:53.361 DEBUG:teuthology.orchestra.run.smithi061:> curl http://smithi061.front.sepia.ceph.com:8000/ 2021-02-11T06:13:53.387 INFO:teuthology.orchestra.run.smithi061.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-11T06:13:53.388 INFO:teuthology.orchestra.run.smithi061.stderr: Dload Upload Total Spent Left Speed 2021-02-11T06:13:53.392 INFO:teuthology.orchestra.run.smithi061.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi061.front.sepia.ceph.com port 8000: Connection refused 2021-02-11T06:13:53.392 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-02-11T06:13:54.394 DEBUG:teuthology.orchestra.run.smithi061:> curl http://smithi061.front.sepia.ceph.com:8000/ 2021-02-11T06:13:54.408 INFO:teuthology.orchestra.run.smithi061.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-11T06:13:54.408 INFO:teuthology.orchestra.run.smithi061.stderr: Dload Upload Total Spent Left Speed 2021-02-11T06:13:54.415 INFO:teuthology.orchestra.run.smithi061.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 30571 0 --:--:-- --:--:-- --:--:-- 30571 2021-02-11T06:13:54.416 INFO:teuthology.orchestra.run.smithi061.stdout:anonymous 2021-02-11T06:13:54.416 INFO:tasks.rgw:Polling c1.client.1 until it starts accepting connections on http://smithi061.front.sepia.ceph.com:8001/ 2021-02-11T06:13:54.417 DEBUG:teuthology.orchestra.run.smithi061:> curl http://smithi061.front.sepia.ceph.com:8001/ 2021-02-11T06:13:54.470 INFO:teuthology.orchestra.run.smithi061.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-11T06:13:54.470 INFO:teuthology.orchestra.run.smithi061.stderr: Dload Upload Total Spent Left Speed 2021-02-11T06:13:54.477 INFO:teuthology.orchestra.run.smithi061.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 35666 0 --:--:-- --:--:-- --:--:-- 35666 2021-02-11T06:13:54.477 INFO:teuthology.orchestra.run.smithi061.stdout:anonymous 2021-02-11T06:13:54.478 INFO:tasks.rgw:Polling c2.client.0 until it starts accepting connections on http://smithi172.front.sepia.ceph.com:8000/ 2021-02-11T06:13:54.478 DEBUG:teuthology.orchestra.run.smithi172:> curl http://smithi172.front.sepia.ceph.com:8000/ 2021-02-11T06:13:54.726 INFO:teuthology.orchestra.run.smithi172.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-11T06:13:54.727 INFO:teuthology.orchestra.run.smithi172.stderr: Dload Upload Total Spent Left Speed 2021-02-11T06:13:54.734 INFO:teuthology.orchestra.run.smithi172.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 30571 0 --:--:-- --:--:-- --:--:-- 30571 2021-02-11T06:13:54.734 INFO:teuthology.orchestra.run.smithi172.stdout:anonymous 2021-02-11T06:13:54.735 INFO:tasks.rgw:Polling c2.client.1 until it starts accepting connections on http://smithi172.front.sepia.ceph.com:8001/ 2021-02-11T06:13:54.735 DEBUG:teuthology.orchestra.run.smithi172:> curl http://smithi172.front.sepia.ceph.com:8001/ 2021-02-11T06:13:54.751 INFO:teuthology.orchestra.run.smithi172.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-11T06:13:54.751 INFO:teuthology.orchestra.run.smithi172.stderr: Dload Upload Total Spent Left Speed 2021-02-11T06:13:54.759 INFO:teuthology.orchestra.run.smithi172.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 30571 0 --:--:-- --:--:-- --:--:-- 30571 2021-02-11T06:13:54.759 INFO:teuthology.orchestra.run.smithi172.stdout:anonymous 2021-02-11T06:13:54.760 INFO:teuthology.run_tasks:Running task rgw-multisite... 2021-02-11T06:13:55.245 INFO:tasks.rgw_multisite:creating realm.. 2021-02-11T06:13:55.245 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:13:55.245 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:13:55.245 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 realm create --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T06:13:55.283 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:13:55.283 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:13:55.334 INFO:teuthology.orchestra.run.smithi061.stdout:{"id":"a797f55e-5ffc-4df8-acfc-1a39dd991822","name":"test-realm","current_period":"1bd98108-bc65-40b0-bdb9-4c115f1e8a91","epoch":1} 2021-02-11T06:13:55.335 INFO:tasks.rgw_multisite:creating master zonegroup.. 2021-02-11T06:13:55.335 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:13:55.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:13:55.336 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup set --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T06:13:55.416 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:13:55.417 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:13:55.441 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:13:55.441+0000 7ff459248080 -1 old format 2021-02-11T06:13:55.461 INFO:teuthology.orchestra.run.smithi061.stdout:{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi061.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"","zones":[],"placement_targets":[],"default_placement":"","realm_id":"a797f55e-5ffc-4df8-acfc-1a39dd991822","sync_policy":{"groups":[]}} 2021-02-11T06:13:55.749 INFO:tasks.rgw_multisite:creating master zone.. 2021-02-11T06:13:55.750 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', '307AEWIWKOCPYEOH', '--secret', 'qEUy5RJVuZgF4tphJ6ujQNFJaaIFkUpT', '--endpoints', 'http://smithi061.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:13:55.750 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', '307AEWIWKOCPYEOH', '--secret', 'qEUy5RJVuZgF4tphJ6ujQNFJaaIFkUpT', '--endpoints', 'http://smithi061.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:13:55.750 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --master --access-key 307AEWIWKOCPYEOH --secret qEUy5RJVuZgF4tphJ6ujQNFJaaIFkUpT --endpoints http://smithi061.front.sepia.ceph.com:8000 --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T06:13:55.799 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:13:55.799 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:13:55.878 INFO:teuthology.orchestra.run.smithi061.stdout:{"id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7","name":"test-zone1","domain_root":"test-zone1.rgw.meta:root","control_pool":"test-zone1.rgw.control","gc_pool":"test-zone1.rgw.log:gc","lc_pool":"test-zone1.rgw.log:lc","log_pool":"test-zone1.rgw.log","intent_log_pool":"test-zone1.rgw.log:intent","usage_log_pool":"test-zone1.rgw.log:usage","roles_pool":"test-zone1.rgw.meta:roles","reshard_pool":"test-zone1.rgw.log:reshard","user_keys_pool":"test-zone1.rgw.meta:users.keys","user_email_pool":"test-zone1.rgw.meta:users.email","user_swift_pool":"test-zone1.rgw.meta:users.swift","user_uid_pool":"test-zone1.rgw.meta:users.uid","otp_pool":"test-zone1.rgw.otp","system_key":{"access_key":"307AEWIWKOCPYEOH","secret_key":"qEUy5RJVuZgF4tphJ6ujQNFJaaIFkUpT"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone1.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone1.rgw.buckets.data"}},"data_extra_pool":"test-zone1.rgw.buckets.non-ec","index_type":0}}],"realm_id":"a797f55e-5ffc-4df8-acfc-1a39dd991822","notif_pool":"test-zone1.rgw.log:notif"} 2021-02-11T06:13:55.878 DEBUG:teuthology.orchestra.run.smithi061:> sudo ceph osd pool create test-zone1.rgw.buckets.data 64 64 --cluster c1 2021-02-11T06:13:56.259 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'test-zone1.rgw.buckets.data' created 2021-02-11T06:13:56.274 DEBUG:teuthology.orchestra.run.smithi061:> sudo ceph osd pool application enable test-zone1.rgw.buckets.data rgw --cluster c1 2021-02-11T06:13:57.283 INFO:teuthology.orchestra.run.smithi061.stderr:enabled application 'rgw' on pool 'test-zone1.rgw.buckets.data' 2021-02-11T06:13:57.304 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:13:57.304 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:13:57.305 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T06:13:57.354 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:13:57.354 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:13:57.379 INFO:teuthology.orchestra.run.smithi061.stdout:{"id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7","name":"test-zone1","domain_root":"test-zone1.rgw.meta:root","control_pool":"test-zone1.rgw.control","gc_pool":"test-zone1.rgw.log:gc","lc_pool":"test-zone1.rgw.log:lc","log_pool":"test-zone1.rgw.log","intent_log_pool":"test-zone1.rgw.log:intent","usage_log_pool":"test-zone1.rgw.log:usage","roles_pool":"test-zone1.rgw.meta:roles","reshard_pool":"test-zone1.rgw.log:reshard","user_keys_pool":"test-zone1.rgw.meta:users.keys","user_email_pool":"test-zone1.rgw.meta:users.email","user_swift_pool":"test-zone1.rgw.meta:users.swift","user_uid_pool":"test-zone1.rgw.meta:users.uid","otp_pool":"test-zone1.rgw.otp","system_key":{"access_key":"307AEWIWKOCPYEOH","secret_key":"qEUy5RJVuZgF4tphJ6ujQNFJaaIFkUpT"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone1.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone1.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone1.rgw.buckets.non-ec","index_type":0}}],"realm_id":"","notif_pool":"test-zone1.rgw.log:notif"} 2021-02-11T06:13:57.380 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:13:57.380 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:13:57.380 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T06:13:57.417 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:13:57.417 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:14:05.556 INFO:teuthology.orchestra.run.smithi061.stdout:{"id":"7c4a812f-81e4-4a72-acce-0ee18589648c","epoch":1,"predecessor_uuid":"1bd98108-bc65-40b0-bdb9-4c115f1e8a91","sync_status":[],"period_map":{"id":"7c4a812f-81e4-4a72-acce-0ee18589648c","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi061.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7","zones":[{"id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7","name":"test-zone1","endpoints":["http://smithi061.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"false","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"a797f55e-5ffc-4df8-acfc-1a39dd991822","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7","val":3877615972}]},"master_zonegroup":"test-zonegroup","master_zone":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"a797f55e-5ffc-4df8-acfc-1a39dd991822","realm_name":"test-realm","realm_epoch":2} 2021-02-11T06:14:05.557 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2021-02-11T06:14:05.557 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T06:14:05.599 INFO:tasks.rgw.c1.client.0.smithi061.stdout:2021-02-11T06:14:05.601+0000 7f1b919c4700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground (PID: 17144) UID: 0 2021-02-11T06:14:05.599 INFO:tasks.rgw.c1.client.0.smithi061.stdout:2021-02-11T06:14:05.601+0000 7f1ba94eb0c0 -1 shutting down 2021-02-11T06:14:10.256 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2021-02-11T06:14:14.791 INFO:tasks.rgw.c1.client.0:Stopped 2021-02-11T06:14:14.792 INFO:tasks.rgw.c1.client.0:Restarting daemon 2021-02-11T06:14:14.792 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2021-02-11T06:14:14.794 INFO:tasks.rgw.c1.client.0:Started 2021-02-11T06:14:14.795 DEBUG:teuthology.orchestra.run.smithi061:> curl http://smithi061.front.sepia.ceph.com:8000 2021-02-11T06:14:14.812 INFO:teuthology.orchestra.run.smithi061.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-11T06:14:14.812 INFO:teuthology.orchestra.run.smithi061.stderr: Dload Upload Total Spent Left Speed 2021-02-11T06:14:14.816 INFO:teuthology.orchestra.run.smithi061.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi061.front.sepia.ceph.com port 8000: Connection refused 2021-02-11T06:14:14.817 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-02-11T06:14:15.818 DEBUG:teuthology.orchestra.run.smithi061:> curl http://smithi061.front.sepia.ceph.com:8000 2021-02-11T06:14:15.833 INFO:teuthology.orchestra.run.smithi061.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-11T06:14:15.833 INFO:teuthology.orchestra.run.smithi061.stderr: Dload Upload Total Spent Left Speed 2021-02-11T06:14:15.840 INFO:teuthology.orchestra.run.smithi061.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 30571 0 --:--:-- --:--:-- --:--:-- 30571 2021-02-11T06:14:15.841 INFO:teuthology.orchestra.run.smithi061.stdout:anonymous 2021-02-11T06:14:15.842 INFO:tasks.rgw_multisite:creating admin user.. 2021-02-11T06:14:15.842 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', '307AEWIWKOCPYEOH', '--secret', 'qEUy5RJVuZgF4tphJ6ujQNFJaaIFkUpT', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:14:15.842 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', '307AEWIWKOCPYEOH', '--secret', 'qEUy5RJVuZgF4tphJ6ujQNFJaaIFkUpT', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:14:15.842 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid realm-admin --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Realm Admin' --system --access-key 307AEWIWKOCPYEOH --secret qEUy5RJVuZgF4tphJ6ujQNFJaaIFkUpT --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T06:14:15.935 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:14:15.935 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:14:16.153 INFO:teuthology.orchestra.run.smithi061.stdout:{"user_id":"realm-admin","display_name":"Realm Admin","email":"","suspended":0,"max_buckets":1000,"subusers":[],"keys":[{"user":"realm-admin","access_key":"307AEWIWKOCPYEOH","secret_key":"qEUy5RJVuZgF4tphJ6ujQNFJaaIFkUpT"}],"swift_keys":[],"caps":[],"op_mask":"read, write, delete","system":"true","default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[]} 2021-02-11T06:14:16.183 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2021-02-11T06:14:16.184 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi061.front.sepia.ceph.com:8000', '--access-key', '307AEWIWKOCPYEOH', '--secret', 'qEUy5RJVuZgF4tphJ6ujQNFJaaIFkUpT', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:14:16.184 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi061.front.sepia.ceph.com:8000', '--access-key', '307AEWIWKOCPYEOH', '--secret', 'qEUy5RJVuZgF4tphJ6ujQNFJaaIFkUpT', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:14:16.184 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi061.front.sepia.ceph.com:8000 --access-key 307AEWIWKOCPYEOH --secret qEUy5RJVuZgF4tphJ6ujQNFJaaIFkUpT --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T06:14:16.228 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:14:16.228 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:14:16.355 INFO:teuthology.orchestra.run.smithi172.stdout:{"id":"a797f55e-5ffc-4df8-acfc-1a39dd991822","name":"test-realm","current_period":"7c4a812f-81e4-4a72-acce-0ee18589648c","epoch":2} 2021-02-11T06:14:16.355 INFO:tasks.rgw_multisite:creating zone.. 2021-02-11T06:14:16.355 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', '307AEWIWKOCPYEOH', '--secret', 'qEUy5RJVuZgF4tphJ6ujQNFJaaIFkUpT', '--endpoints', 'http://smithi172.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:14:16.356 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', '307AEWIWKOCPYEOH', '--secret', 'qEUy5RJVuZgF4tphJ6ujQNFJaaIFkUpT', '--endpoints', 'http://smithi172.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:14:16.356 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --access-key 307AEWIWKOCPYEOH --secret qEUy5RJVuZgF4tphJ6ujQNFJaaIFkUpT --endpoints http://smithi172.front.sepia.ceph.com:8000 --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T06:14:16.439 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:14:16.439 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:14:16.501 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:14:16.504+0000 7f6a9f8e3080 0 failed reading obj info from .rgw.root:zone_info.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7: (2) No such file or directory 2021-02-11T06:14:16.502 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:14:16.504+0000 7f6a9f8e3080 0 WARNING: could not read zone params for zone id=5f91a5b1-d54f-4a3d-a100-b2911a08a2e7 name=test-zone1 2021-02-11T06:14:16.516 INFO:teuthology.orchestra.run.smithi172.stdout:{"id":"e4b3955d-973b-4dd2-ae54-b7e681fbeb21","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"307AEWIWKOCPYEOH","secret_key":"qEUy5RJVuZgF4tphJ6ujQNFJaaIFkUpT"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"a797f55e-5ffc-4df8-acfc-1a39dd991822","notif_pool":"test-zone2.rgw.log:notif"} 2021-02-11T06:14:16.516 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph osd pool create test-zone2.rgw.buckets.data 64 64 --cluster c2 2021-02-11T06:14:17.186 INFO:teuthology.orchestra.run.smithi172.stderr:pool 'test-zone2.rgw.buckets.data' created 2021-02-11T06:14:17.206 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph osd pool application enable test-zone2.rgw.buckets.data rgw --cluster c2 2021-02-11T06:14:18.202 INFO:teuthology.orchestra.run.smithi172.stderr:enabled application 'rgw' on pool 'test-zone2.rgw.buckets.data' 2021-02-11T06:14:18.222 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:14:18.222 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:14:18.222 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T06:14:18.268 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:14:18.268 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:14:18.293 INFO:teuthology.orchestra.run.smithi172.stdout:{"id":"e4b3955d-973b-4dd2-ae54-b7e681fbeb21","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"307AEWIWKOCPYEOH","secret_key":"qEUy5RJVuZgF4tphJ6ujQNFJaaIFkUpT"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"","notif_pool":"test-zone2.rgw.log:notif"} 2021-02-11T06:14:18.294 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:14:18.294 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:14:18.294 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T06:14:18.372 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:14:18.372 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:14:20.492 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2021-02-11T06:14:26.408 INFO:teuthology.orchestra.run.smithi172.stderr:Sending period to new master zone 5f91a5b1-d54f-4a3d-a100-b2911a08a2e7 2021-02-11T06:14:26.507 INFO:teuthology.orchestra.run.smithi172.stdout:{"id":"7c4a812f-81e4-4a72-acce-0ee18589648c","epoch":2,"predecessor_uuid":"1bd98108-bc65-40b0-bdb9-4c115f1e8a91","sync_status":[],"period_map":{"id":"7c4a812f-81e4-4a72-acce-0ee18589648c","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi061.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7","zones":[{"id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7","name":"test-zone1","endpoints":["http://smithi061.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"e4b3955d-973b-4dd2-ae54-b7e681fbeb21","name":"test-zone2","endpoints":["http://smithi172.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"a797f55e-5ffc-4df8-acfc-1a39dd991822","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7","val":3877615972},{"key":"e4b3955d-973b-4dd2-ae54-b7e681fbeb21","val":3820998694}]},"master_zonegroup":"test-zonegroup","master_zone":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"a797f55e-5ffc-4df8-acfc-1a39dd991822","realm_name":"test-realm","realm_epoch":2} 2021-02-11T06:14:26.508 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2021-02-11T06:14:26.508 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T06:14:26.549 INFO:tasks.rgw.c2.client.0.smithi172.stdout:2021-02-11T06:14:26.552+0000 7fbf12ffd700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground (PID: 17167) UID: 0 2021-02-11T06:14:26.549 INFO:tasks.rgw.c2.client.0.smithi172.stdout:2021-02-11T06:14:26.552+0000 7fbf2f1880c0 -1 shutting down 2021-02-11T06:14:28.950 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2021-02-11T06:14:34.632 INFO:tasks.rgw.c2.client.0:Stopped 2021-02-11T06:14:34.633 INFO:tasks.rgw.c2.client.0:Restarting daemon 2021-02-11T06:14:34.633 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2021-02-11T06:14:34.636 INFO:tasks.rgw.c2.client.0:Started 2021-02-11T06:14:34.636 DEBUG:teuthology.orchestra.run.smithi172:> curl http://smithi172.front.sepia.ceph.com:8000 2021-02-11T06:14:34.653 INFO:teuthology.orchestra.run.smithi172.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-11T06:14:34.654 INFO:teuthology.orchestra.run.smithi172.stderr: Dload Upload Total Spent Left Speed 2021-02-11T06:14:34.658 INFO:teuthology.orchestra.run.smithi172.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi172.front.sepia.ceph.com port 8000: Connection refused 2021-02-11T06:14:34.658 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-02-11T06:14:35.659 DEBUG:teuthology.orchestra.run.smithi172:> curl http://smithi172.front.sepia.ceph.com:8000 2021-02-11T06:14:35.672 INFO:teuthology.orchestra.run.smithi172.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-11T06:14:35.672 INFO:teuthology.orchestra.run.smithi172.stderr: Dload Upload Total Spent Left Speed 2021-02-11T06:14:35.679 INFO:teuthology.orchestra.run.smithi172.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 30571 0 --:--:-- --:--:-- --:--:-- 26750 2021-02-11T06:14:35.679 INFO:teuthology.orchestra.run.smithi172.stdout:anonymous 2021-02-11T06:14:35.680 INFO:tasks.rgw_multisite:creating zone.. 2021-02-11T06:14:35.680 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', '307AEWIWKOCPYEOH', '--secret', 'qEUy5RJVuZgF4tphJ6ujQNFJaaIFkUpT', '--endpoints', 'http://smithi061.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:14:35.680 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', '307AEWIWKOCPYEOH', '--secret', 'qEUy5RJVuZgF4tphJ6ujQNFJaaIFkUpT', '--endpoints', 'http://smithi061.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:14:35.680 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key 307AEWIWKOCPYEOH --secret qEUy5RJVuZgF4tphJ6ujQNFJaaIFkUpT --endpoints http://smithi061.front.sepia.ceph.com:8001 --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T06:14:35.727 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:14:35.727 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:14:35.790 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:14:35.793+0000 7eff56944080 0 failed reading obj info from .rgw.root:zone_info.e4b3955d-973b-4dd2-ae54-b7e681fbeb21: (2) No such file or directory 2021-02-11T06:14:35.791 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:14:35.793+0000 7eff56944080 0 WARNING: could not read zone params for zone id=e4b3955d-973b-4dd2-ae54-b7e681fbeb21 name=test-zone2 2021-02-11T06:14:35.800 INFO:teuthology.orchestra.run.smithi061.stdout:{"id":"bca56477-0808-4ebe-8649-b1169fc76383","name":"test-zone3","domain_root":"test-zone3.rgw.meta:root","control_pool":"test-zone3.rgw.control","gc_pool":"test-zone3.rgw.log:gc","lc_pool":"test-zone3.rgw.log:lc","log_pool":"test-zone3.rgw.log","intent_log_pool":"test-zone3.rgw.log:intent","usage_log_pool":"test-zone3.rgw.log:usage","roles_pool":"test-zone3.rgw.meta:roles","reshard_pool":"test-zone3.rgw.log:reshard","user_keys_pool":"test-zone3.rgw.meta:users.keys","user_email_pool":"test-zone3.rgw.meta:users.email","user_swift_pool":"test-zone3.rgw.meta:users.swift","user_uid_pool":"test-zone3.rgw.meta:users.uid","otp_pool":"test-zone3.rgw.otp","system_key":{"access_key":"307AEWIWKOCPYEOH","secret_key":"qEUy5RJVuZgF4tphJ6ujQNFJaaIFkUpT"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone3.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone3.rgw.buckets.data"}},"data_extra_pool":"test-zone3.rgw.buckets.non-ec","index_type":0}}],"realm_id":"a797f55e-5ffc-4df8-acfc-1a39dd991822","notif_pool":"test-zone3.rgw.log:notif"} 2021-02-11T06:14:35.801 DEBUG:teuthology.orchestra.run.smithi061:> sudo ceph osd pool create test-zone3.rgw.buckets.data 64 64 --cluster c1 2021-02-11T06:14:37.105 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'test-zone3.rgw.buckets.data' created 2021-02-11T06:14:37.129 DEBUG:teuthology.orchestra.run.smithi061:> sudo ceph osd pool application enable test-zone3.rgw.buckets.data rgw --cluster c1 2021-02-11T06:14:38.125 INFO:teuthology.orchestra.run.smithi061.stderr:enabled application 'rgw' on pool 'test-zone3.rgw.buckets.data' 2021-02-11T06:14:38.152 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:14:38.152 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:14:38.152 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T06:14:38.191 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:14:38.191 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:14:38.217 INFO:teuthology.orchestra.run.smithi061.stdout:{"id":"bca56477-0808-4ebe-8649-b1169fc76383","name":"test-zone3","domain_root":"test-zone3.rgw.meta:root","control_pool":"test-zone3.rgw.control","gc_pool":"test-zone3.rgw.log:gc","lc_pool":"test-zone3.rgw.log:lc","log_pool":"test-zone3.rgw.log","intent_log_pool":"test-zone3.rgw.log:intent","usage_log_pool":"test-zone3.rgw.log:usage","roles_pool":"test-zone3.rgw.meta:roles","reshard_pool":"test-zone3.rgw.log:reshard","user_keys_pool":"test-zone3.rgw.meta:users.keys","user_email_pool":"test-zone3.rgw.meta:users.email","user_swift_pool":"test-zone3.rgw.meta:users.swift","user_uid_pool":"test-zone3.rgw.meta:users.uid","otp_pool":"test-zone3.rgw.otp","system_key":{"access_key":"307AEWIWKOCPYEOH","secret_key":"qEUy5RJVuZgF4tphJ6ujQNFJaaIFkUpT"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone3.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone3.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone3.rgw.buckets.non-ec","index_type":0}}],"realm_id":"","notif_pool":"test-zone3.rgw.log:notif"} 2021-02-11T06:14:38.217 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:14:38.217 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:14:38.217 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T06:14:38.296 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:14:38.297 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:14:40.335 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2021-02-11T06:14:46.337 INFO:teuthology.orchestra.run.smithi061.stderr:Sending period to new master zone 5f91a5b1-d54f-4a3d-a100-b2911a08a2e7 2021-02-11T06:14:46.430 INFO:teuthology.orchestra.run.smithi061.stdout:{"id":"7c4a812f-81e4-4a72-acce-0ee18589648c","epoch":3,"predecessor_uuid":"1bd98108-bc65-40b0-bdb9-4c115f1e8a91","sync_status":[],"period_map":{"id":"7c4a812f-81e4-4a72-acce-0ee18589648c","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi061.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7","zones":[{"id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7","name":"test-zone1","endpoints":["http://smithi061.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"bca56477-0808-4ebe-8649-b1169fc76383","name":"test-zone3","endpoints":["http://smithi061.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"e4b3955d-973b-4dd2-ae54-b7e681fbeb21","name":"test-zone2","endpoints":["http://smithi172.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"a797f55e-5ffc-4df8-acfc-1a39dd991822","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7","val":3877615972},{"key":"bca56477-0808-4ebe-8649-b1169fc76383","val":2043214290},{"key":"e4b3955d-973b-4dd2-ae54-b7e681fbeb21","val":3820998694}]},"master_zonegroup":"test-zonegroup","master_zone":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"a797f55e-5ffc-4df8-acfc-1a39dd991822","realm_name":"test-realm","realm_epoch":2} 2021-02-11T06:14:46.431 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2021-02-11T06:14:46.431 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T06:14:46.474 INFO:tasks.rgw.c1.client.1.smithi061.stdout:2021-02-11T06:14:46.477+0000 7fbddeffd700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground (PID: 17147) UID: 0 2021-02-11T06:14:46.474 INFO:tasks.rgw.c1.client.1.smithi061.stdout:2021-02-11T06:14:46.477+0000 7fbdfb0d50c0 -1 shutting down 2021-02-11T06:14:51.650 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2021-02-11T06:14:56.699 INFO:tasks.rgw.c1.client.1:Stopped 2021-02-11T06:14:56.699 INFO:tasks.rgw.c1.client.1:Restarting daemon 2021-02-11T06:14:56.700 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2021-02-11T06:14:56.702 INFO:tasks.rgw.c1.client.1:Started 2021-02-11T06:14:56.702 DEBUG:teuthology.orchestra.run.smithi061:> curl http://smithi061.front.sepia.ceph.com:8001 2021-02-11T06:14:56.720 INFO:teuthology.orchestra.run.smithi061.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-11T06:14:56.720 INFO:teuthology.orchestra.run.smithi061.stderr: Dload Upload Total Spent Left Speed 2021-02-11T06:14:56.724 INFO:teuthology.orchestra.run.smithi061.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi061.front.sepia.ceph.com port 8001: Connection refused 2021-02-11T06:14:56.724 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-02-11T06:14:57.726 DEBUG:teuthology.orchestra.run.smithi061:> curl http://smithi061.front.sepia.ceph.com:8001 2021-02-11T06:14:57.739 INFO:teuthology.orchestra.run.smithi061.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-11T06:14:57.739 INFO:teuthology.orchestra.run.smithi061.stderr: Dload Upload Total Spent Left Speed 2021-02-11T06:14:57.746 INFO:teuthology.orchestra.run.smithi061.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 30571 0 --:--:-- --:--:-- --:--:-- 30571 2021-02-11T06:14:57.747 INFO:teuthology.orchestra.run.smithi061.stdout:anonymous 2021-02-11T06:14:57.747 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2021-02-11T06:14:57.748 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi061.front.sepia.ceph.com:8000', '--access-key', '307AEWIWKOCPYEOH', '--secret', 'qEUy5RJVuZgF4tphJ6ujQNFJaaIFkUpT', '--url', 'http://smithi061.front.sepia.ceph.com:8000', '--access-key', '307AEWIWKOCPYEOH', '--secret', 'qEUy5RJVuZgF4tphJ6ujQNFJaaIFkUpT', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:14:57.748 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi061.front.sepia.ceph.com:8000', '--access-key', '307AEWIWKOCPYEOH', '--secret', 'qEUy5RJVuZgF4tphJ6ujQNFJaaIFkUpT', '--url', 'http://smithi061.front.sepia.ceph.com:8000', '--access-key', '307AEWIWKOCPYEOH', '--secret', 'qEUy5RJVuZgF4tphJ6ujQNFJaaIFkUpT', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:14:57.748 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi061.front.sepia.ceph.com:8000 --access-key 307AEWIWKOCPYEOH --secret qEUy5RJVuZgF4tphJ6ujQNFJaaIFkUpT --url http://smithi061.front.sepia.ceph.com:8000 --access-key 307AEWIWKOCPYEOH --secret qEUy5RJVuZgF4tphJ6ujQNFJaaIFkUpT --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T06:14:57.798 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:14:57.799 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:14:57.849 INFO:tasks.rgw.c2.client.0.smithi172.stdout:2021-02-11T06:14:57.852+0000 7fb2cdffb700 -1 rgw realm watcher: RGWRealmWatcher::handle_error oid=realms.a797f55e-5ffc-4df8-acfc-1a39dd991822.control err=-107 2021-02-11T06:14:57.879 INFO:teuthology.orchestra.run.smithi172.stdout:{"id":"a797f55e-5ffc-4df8-acfc-1a39dd991822","name":"test-realm","current_period":"7c4a812f-81e4-4a72-acce-0ee18589648c","epoch":2} 2021-02-11T06:14:57.879 INFO:tasks.rgw_multisite:creating zone.. 2021-02-11T06:14:57.879 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', '307AEWIWKOCPYEOH', '--secret', 'qEUy5RJVuZgF4tphJ6ujQNFJaaIFkUpT', '--endpoints', 'http://smithi172.front.sepia.ceph.com:8001', '--tier-type', 'pubsub', '--sync-from-all=0', '--sync-from', 'test-zone1', '--tier-config', 'start_with_full_sync=false,event_retention_days=7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:14:57.879 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', '307AEWIWKOCPYEOH', '--secret', 'qEUy5RJVuZgF4tphJ6ujQNFJaaIFkUpT', '--endpoints', 'http://smithi172.front.sepia.ceph.com:8001', '--tier-type', 'pubsub', '--sync-from-all=0', '--sync-from', 'test-zone1', '--tier-config', 'start_with_full_sync=false,event_retention_days=7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:14:57.880 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key 307AEWIWKOCPYEOH --secret qEUy5RJVuZgF4tphJ6ujQNFJaaIFkUpT --endpoints http://smithi172.front.sepia.ceph.com:8001 --tier-type pubsub --sync-from-all=0 --sync-from test-zone1 --tier-config start_with_full_sync=false,event_retention_days=7 --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T06:14:57.932 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:14:57.932 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:14:57.998 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:14:58.000+0000 7fc6f50c6080 0 failed reading obj info from .rgw.root:zone_info.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7: (2) No such file or directory 2021-02-11T06:14:57.998 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:14:58.000+0000 7fc6f50c6080 0 WARNING: could not read zone params for zone id=5f91a5b1-d54f-4a3d-a100-b2911a08a2e7 name=test-zone1 2021-02-11T06:14:57.998 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:14:58.000+0000 7fc6f50c6080 0 failed reading obj info from .rgw.root:zone_info.bca56477-0808-4ebe-8649-b1169fc76383: (2) No such file or directory 2021-02-11T06:14:57.998 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:14:58.000+0000 7fc6f50c6080 0 WARNING: could not read zone params for zone id=bca56477-0808-4ebe-8649-b1169fc76383 name=test-zone3 2021-02-11T06:14:58.010 INFO:teuthology.orchestra.run.smithi172.stdout:{"id":"d7ed052a-b089-4bac-932c-80450ae3dd3d","name":"test-zone4","domain_root":"test-zone4.rgw.meta:root","control_pool":"test-zone4.rgw.control","gc_pool":"test-zone4.rgw.log:gc","lc_pool":"test-zone4.rgw.log:lc","log_pool":"test-zone4.rgw.log","intent_log_pool":"test-zone4.rgw.log:intent","usage_log_pool":"test-zone4.rgw.log:usage","roles_pool":"test-zone4.rgw.meta:roles","reshard_pool":"test-zone4.rgw.log:reshard","user_keys_pool":"test-zone4.rgw.meta:users.keys","user_email_pool":"test-zone4.rgw.meta:users.email","user_swift_pool":"test-zone4.rgw.meta:users.swift","user_uid_pool":"test-zone4.rgw.meta:users.uid","otp_pool":"test-zone4.rgw.otp","system_key":{"access_key":"307AEWIWKOCPYEOH","secret_key":"qEUy5RJVuZgF4tphJ6ujQNFJaaIFkUpT"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone4.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone4.rgw.buckets.data"}},"data_extra_pool":"test-zone4.rgw.buckets.non-ec","index_type":0}}],"tier_config":{"event_retention_days":7,"start_with_full_sync":false},"realm_id":"a797f55e-5ffc-4df8-acfc-1a39dd991822","notif_pool":"test-zone4.rgw.log:notif"} 2021-02-11T06:14:58.011 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph osd pool create test-zone4.rgw.buckets.data 64 64 --cluster c2 2021-02-11T06:14:59.185 INFO:teuthology.orchestra.run.smithi172.stderr:pool 'test-zone4.rgw.buckets.data' created 2021-02-11T06:15:14.120 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph osd pool application enable test-zone4.rgw.buckets.data rgw --cluster c2 2021-02-11T06:15:14.823 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2021-02-11T06:15:15.209 INFO:teuthology.orchestra.run.smithi172.stderr:enabled application 'rgw' on pool 'test-zone4.rgw.buckets.data' 2021-02-11T06:15:15.221 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:15:15.221 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:15:15.222 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T06:15:15.258 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:15:15.259 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:15:15.314 INFO:teuthology.orchestra.run.smithi172.stdout:{"id":"d7ed052a-b089-4bac-932c-80450ae3dd3d","name":"test-zone4","domain_root":"test-zone4.rgw.meta:root","control_pool":"test-zone4.rgw.control","gc_pool":"test-zone4.rgw.log:gc","lc_pool":"test-zone4.rgw.log:lc","log_pool":"test-zone4.rgw.log","intent_log_pool":"test-zone4.rgw.log:intent","usage_log_pool":"test-zone4.rgw.log:usage","roles_pool":"test-zone4.rgw.meta:roles","reshard_pool":"test-zone4.rgw.log:reshard","user_keys_pool":"test-zone4.rgw.meta:users.keys","user_email_pool":"test-zone4.rgw.meta:users.email","user_swift_pool":"test-zone4.rgw.meta:users.swift","user_uid_pool":"test-zone4.rgw.meta:users.uid","otp_pool":"test-zone4.rgw.otp","system_key":{"access_key":"307AEWIWKOCPYEOH","secret_key":"qEUy5RJVuZgF4tphJ6ujQNFJaaIFkUpT"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone4.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone4.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone4.rgw.buckets.non-ec","index_type":0}}],"tier_config":{"event_retention_days":7,"start_with_full_sync":false},"realm_id":"","notif_pool":"test-zone4.rgw.log:notif"} 2021-02-11T06:15:15.315 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:15:15.315 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:15:15.315 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T06:15:15.398 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:15:15.399 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:15:23.416 INFO:teuthology.orchestra.run.smithi172.stderr:Sending period to new master zone 5f91a5b1-d54f-4a3d-a100-b2911a08a2e7 2021-02-11T06:15:23.516 INFO:teuthology.orchestra.run.smithi172.stdout:{"id":"7c4a812f-81e4-4a72-acce-0ee18589648c","epoch":4,"predecessor_uuid":"1bd98108-bc65-40b0-bdb9-4c115f1e8a91","sync_status":[],"period_map":{"id":"7c4a812f-81e4-4a72-acce-0ee18589648c","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi061.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7","zones":[{"id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7","name":"test-zone1","endpoints":["http://smithi061.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"bca56477-0808-4ebe-8649-b1169fc76383","name":"test-zone3","endpoints":["http://smithi061.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"d7ed052a-b089-4bac-932c-80450ae3dd3d","name":"test-zone4","endpoints":["http://smithi172.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"e4b3955d-973b-4dd2-ae54-b7e681fbeb21","name":"test-zone2","endpoints":["http://smithi172.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"a797f55e-5ffc-4df8-acfc-1a39dd991822","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7","val":3877615972},{"key":"bca56477-0808-4ebe-8649-b1169fc76383","val":2043214290},{"key":"d7ed052a-b089-4bac-932c-80450ae3dd3d","val":3210761346},{"key":"e4b3955d-973b-4dd2-ae54-b7e681fbeb21","val":3820998694}]},"master_zonegroup":"test-zonegroup","master_zone":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"a797f55e-5ffc-4df8-acfc-1a39dd991822","realm_name":"test-realm","realm_epoch":2} 2021-02-11T06:15:23.517 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2021-02-11T06:15:23.517 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T06:15:23.560 INFO:tasks.rgw.c2.client.1.smithi172.stdout:2021-02-11T06:15:23.564+0000 7f18137fe700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground (PID: 17164) UID: 0 2021-02-11T06:15:23.560 INFO:tasks.rgw.c2.client.1.smithi172.stdout:2021-02-11T06:15:23.564+0000 7f1837a770c0 -1 shutting down 2021-02-11T06:15:26.133 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2021-02-11T06:15:33.641 INFO:tasks.rgw.c2.client.1:Stopped 2021-02-11T06:15:33.642 INFO:tasks.rgw.c2.client.1:Restarting daemon 2021-02-11T06:15:33.642 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2021-02-11T06:15:33.645 INFO:tasks.rgw.c2.client.1:Started 2021-02-11T06:15:33.645 DEBUG:teuthology.orchestra.run.smithi172:> curl http://smithi172.front.sepia.ceph.com:8001 2021-02-11T06:15:33.662 INFO:teuthology.orchestra.run.smithi172.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-11T06:15:33.662 INFO:teuthology.orchestra.run.smithi172.stderr: Dload Upload Total Spent Left Speed 2021-02-11T06:15:33.666 INFO:teuthology.orchestra.run.smithi172.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi172.front.sepia.ceph.com port 8001: Connection refused 2021-02-11T06:15:33.666 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-02-11T06:15:34.667 DEBUG:teuthology.orchestra.run.smithi172:> curl http://smithi172.front.sepia.ceph.com:8001 2021-02-11T06:15:34.814 INFO:teuthology.orchestra.run.smithi172.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-11T06:15:34.814 INFO:teuthology.orchestra.run.smithi172.stderr: Dload Upload Total Spent Left Speed 2021-02-11T06:15:34.819 INFO:teuthology.orchestra.run.smithi172.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 134 100 134 0 0 33500 0 --:--:-- --:--:-- --:--:-- 33500 2021-02-11T06:15:34.820 INFO:teuthology.orchestra.run.smithi172.stdout:{"Code":"MethodNotAllowed","RequestId":"tx000000000000000000001-006024cb86-10e2-test-zone4","HostId":"10e2-test-zone4-test-zonegroup"} 2021-02-11T06:15:34.820 INFO:tasks.rgw_multisite:rgw multisite configuration completed 2021-02-11T06:15:34.820 INFO:teuthology.run_tasks:Running task rgw-multisite-tests... 2021-02-11T06:15:34.913 INFO:tasks.rgw_multisite_tests:creating test user.. 2021-02-11T06:15:34.914 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:15:34.914 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:15:34.914 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid rgw-multisite-test-user --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Multisite Test User' --gen-access-key --gen-secret --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T06:15:34.959 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:15:34.959 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:15:35.185 INFO:teuthology.orchestra.run.smithi061.stdout:{"user_id":"rgw-multisite-test-user","display_name":"Multisite Test User","email":"","suspended":0,"max_buckets":1000,"subusers":[],"keys":[{"user":"rgw-multisite-test-user","access_key":"754M3UOP95SKHB48ZI8X","secret_key":"ChiY503pG1UMkIKCHUAXSjtbzlk4X5rs92SuCZli"}],"swift_keys":[],"caps":[],"op_mask":"read, write, delete","default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[]} 2021-02-11T06:15:35.215 INFO:rgw_multi.tests:meta checkpoint 2021-02-11T06:15:35.215 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:15:35.215 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:15:35.215 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:15:35.296 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:15:35.297 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:15:35.509 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=7c4a812f-81e4-4a72-acce-0ee18589648c 2021-02-11T06:15:35.544 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613024135.181851_48.1","last_update":"2021-02-11T06:15:35.181851Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:15:35.545 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T06:15:35.545 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:15:35.545 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:15:35.545 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:15:35.584 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:15:35.584 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:15:35.823 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7c4a812f-81e4-4a72-acce-0ee18589648c","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:15:35.823 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613024135.181851_48.1 target= 2021-02-11T06:15:39.342 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 has been restored 2021-02-11T06:15:40.825 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:15:40.825 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:15:40.825 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:15:40.866 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:15:40.866 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:15:41.110 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7c4a812f-81e4-4a72-acce-0ee18589648c","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:15:41.110 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613024135.181851_48.1 target= 2021-02-11T06:15:46.112 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:15:46.112 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:15:46.112 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:15:46.152 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:15:46.153 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:15:46.409 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7c4a812f-81e4-4a72-acce-0ee18589648c","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:15:46.409 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613024135.181851_48.1 target= 2021-02-11T06:15:51.410 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:15:51.411 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:15:51.411 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:15:51.451 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:15:51.452 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:15:51.702 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7c4a812f-81e4-4a72-acce-0ee18589648c","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:15:51.702 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613024135.181851_48.1 target= 2021-02-11T06:15:56.704 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:16:05.040 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:16:05.041 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:16:05.083 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:16:05.083 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:16:05.335 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7c4a812f-81e4-4a72-acce-0ee18589648c","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:16:05.336 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613024135.181851_48.1 target= 2021-02-11T06:16:10.336 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:16:10.337 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:16:10.337 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:16:10.377 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:16:10.378 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:16:10.627 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7c4a812f-81e4-4a72-acce-0ee18589648c","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:16:10.628 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613024135.181851_48.1 target= 2021-02-11T06:16:15.629 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:16:31.516 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:16:31.516 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:16:31.558 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:16:31.558 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:16:31.826 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7c4a812f-81e4-4a72-acce-0ee18589648c","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:16:31.826 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613024135.181851_48.1 target= 2021-02-11T06:16:36.827 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:16:36.827 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:16:36.828 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:16:36.870 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:16:36.871 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:16:37.109 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7c4a812f-81e4-4a72-acce-0ee18589648c","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:16:37.109 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613024135.181851_48.1 target= 2021-02-11T06:16:42.110 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:16:42.110 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:16:42.111 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:16:42.152 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:16:42.152 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:16:42.388 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7c4a812f-81e4-4a72-acce-0ee18589648c","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:16:42.389 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613024135.181851_48.1 target= 2021-02-11T06:16:47.389 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:16:53.918 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:16:53.919 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:16:53.959 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:16:53.959 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:16:54.213 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7c4a812f-81e4-4a72-acce-0ee18589648c","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:16:54.213 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613024135.181851_48.1 target= 2021-02-11T06:16:59.214 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:16:59.215 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:16:59.215 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:16:59.252 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:16:59.252 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:16:59.493 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7c4a812f-81e4-4a72-acce-0ee18589648c","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613024135.181851_48.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:15:35.181851Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:16:59.493 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T06:16:59.493 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T06:16:59.493 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:16:59.494 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:16:59.494 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:16:59.534 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:16:59.534 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:16:59.779 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7c4a812f-81e4-4a72-acce-0ee18589648c","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:16:59.779 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=0 master=1_1613024135.181851_48.1 target= 2021-02-11T06:17:04.780 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:17:04.780 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:17:04.781 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:17:04.821 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:17:04.821 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:17:05.058 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7c4a812f-81e4-4a72-acce-0ee18589648c","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:17:05.059 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=0 master=1_1613024135.181851_48.1 target= 2021-02-11T06:17:10.060 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:17:13.152 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:17:13.153 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:17:13.192 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:17:13.192 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:17:13.455 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7c4a812f-81e4-4a72-acce-0ee18589648c","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:17:13.455 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=0 master=1_1613024135.181851_48.1 target= 2021-02-11T06:17:18.456 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:17:18.456 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:17:18.457 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:17:18.499 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:17:18.500 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:17:18.762 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7c4a812f-81e4-4a72-acce-0ee18589648c","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:17:18.763 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=0 master=1_1613024135.181851_48.1 target= 2021-02-11T06:17:23.764 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:17:23.764 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:17:23.764 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:17:23.805 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:17:23.805 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:17:24.049 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7c4a812f-81e4-4a72-acce-0ee18589648c","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:17:24.050 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=0 master=1_1613024135.181851_48.1 target= 2021-02-11T06:17:29.051 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:17:36.504 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:17:36.505 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:17:36.549 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:17:36.550 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:17:36.806 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7c4a812f-81e4-4a72-acce-0ee18589648c","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613024135.181851_48.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:15:35.181851Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:17:36.807 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T06:17:36.807 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:17:36.807 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:17:36.807 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:17:36.807 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:17:36.847 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:17:36.847 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:17:37.095 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7c4a812f-81e4-4a72-acce-0ee18589648c","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613024135.181851_48.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:15:35.181851Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:17:37.096 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:17:37.096 INFO:rgw_multi.tests:meta checkpoint 2021-02-11T06:17:37.096 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:17:37.096 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:17:37.097 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:17:37.138 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:17:37.138 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:17:37.374 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=7c4a812f-81e4-4a72-acce-0ee18589648c 2021-02-11T06:17:37.405 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613024135.181851_48.1","last_update":"2021-02-11T06:15:35.181851Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:17:37.406 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T06:17:37.406 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:17:37.406 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:17:37.406 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:17:37.449 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:17:37.449 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:17:37.695 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7c4a812f-81e4-4a72-acce-0ee18589648c","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613024135.181851_48.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:15:35.181851Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:17:37.695 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T06:17:37.695 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T06:17:37.695 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:17:37.696 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:17:37.696 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:17:37.739 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:17:37.739 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:17:37.992 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7c4a812f-81e4-4a72-acce-0ee18589648c","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613024135.181851_48.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:15:35.181851Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:17:37.993 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T06:17:37.993 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:17:37.993 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:17:37.993 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:17:37.993 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:17:38.035 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:17:38.035 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:17:38.302 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7c4a812f-81e4-4a72-acce-0ee18589648c","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613024135.181851_48.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:15:35.181851Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:17:38.303 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:17:38.303 INFO:tasks.rgw_multisite_tests:running rgw multisite tests on 'tasks.rgw_multi.tests' with args=[] 2021-02-11T06:17:38.316 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xscaju-1 2021-02-11T06:17:42.091 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xscaju-2 2021-02-11T06:17:45.288 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xscaju-3 2021-02-11T06:17:45.388 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:17:58.237 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:17:58.238 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:17:58.281 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:17:58.282 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:17:58.508 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=7c4a812f-81e4-4a72-acce-0ee18589648c 2021-02-11T06:17:58.549 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613024135.181851_48.1","last_update":"2021-02-11T06:15:35.181851Z"},{"marker":"1_1613024265.357008_40.1","last_update":"2021-02-11T06:17:45.357008Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:17:58.550 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T06:17:58.550 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:17:58.550 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:17:58.551 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:17:58.592 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:17:58.592 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:17:58.843 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7c4a812f-81e4-4a72-acce-0ee18589648c","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613024135.181851_48.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:15:35.181851Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613024265.357008_40.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:17:45.357008Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:17:58.843 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T06:17:58.844 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T06:17:58.844 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:17:58.844 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:17:58.844 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:17:58.886 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:17:58.886 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:17:59.146 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7c4a812f-81e4-4a72-acce-0ee18589648c","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613024135.181851_48.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:15:35.181851Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613024265.357008_40.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:17:45.357008Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:17:59.147 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T06:17:59.147 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:17:59.147 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:17:59.147 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:17:59.148 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:17:59.188 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:17:59.188 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:17:59.432 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7c4a812f-81e4-4a72-acce-0ee18589648c","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613024135.181851_48.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:15:35.181851Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613024265.357008_40.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:17:45.357008Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:17:59.432 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:17:59.473 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_create ... ok 2021-02-11T06:17:59.474 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xscaju-4 2021-02-11T06:17:59.503 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xscaju-5 2021-02-11T06:17:59.564 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xscaju-6 2021-02-11T06:17:59.631 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:17:59.631 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:17:59.631 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:17:59.667 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:17:59.667 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:17:59.892 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=7c4a812f-81e4-4a72-acce-0ee18589648c 2021-02-11T06:17:59.922 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613024135.181851_48.1","last_update":"2021-02-11T06:15:35.181851Z"},{"marker":"1_1613024279.610246_53.1","last_update":"2021-02-11T06:17:59.610246Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:17:59.923 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T06:17:59.923 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:17:59.923 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:17:59.923 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:17:59.956 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:17:59.957 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:18:00.189 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7c4a812f-81e4-4a72-acce-0ee18589648c","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613024135.181851_48.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:15:35.181851Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613024279.610246_53.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:17:59.610246Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:18:00.190 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T06:18:00.190 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T06:18:00.190 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:18:00.190 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:18:00.190 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:18:00.233 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:18:00.234 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:18:00.492 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7c4a812f-81e4-4a72-acce-0ee18589648c","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613024135.181851_48.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:15:35.181851Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613024279.610246_53.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:17:59.610246Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:18:00.492 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T06:18:00.492 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:18:00.492 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:18:00.493 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:18:00.493 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:18:00.532 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:18:00.532 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:18:00.795 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7c4a812f-81e4-4a72-acce-0ee18589648c","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613024135.181851_48.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:15:35.181851Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613024279.610246_53.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:17:59.610246Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:18:00.795 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:18:01.146 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:18:01.146 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:18:01.146 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:18:01.180 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:18:01.180 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:18:01.404 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=7c4a812f-81e4-4a72-acce-0ee18589648c 2021-02-11T06:18:01.445 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613024135.181851_48.1","last_update":"2021-02-11T06:15:35.181851Z"},{"marker":"1_1613024279.610246_53.1","last_update":"2021-02-11T06:17:59.610246Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:18:01.446 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T06:18:01.446 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:18:01.446 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:18:01.446 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:18:01.487 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:18:01.488 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:18:01.750 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7c4a812f-81e4-4a72-acce-0ee18589648c","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613024135.181851_48.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:15:35.181851Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613024279.610246_53.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:17:59.610246Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:18:01.751 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T06:18:01.751 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T06:18:01.751 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:18:01.751 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:18:01.752 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:18:01.791 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:18:01.791 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:18:02.069 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7c4a812f-81e4-4a72-acce-0ee18589648c","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613024135.181851_48.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:15:35.181851Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613024279.610246_53.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:17:59.610246Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:18:13.764 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T06:18:13.764 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:18:13.764 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:18:13.765 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:18:13.765 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:18:13.808 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:18:13.808 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:18:14.163 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7c4a812f-81e4-4a72-acce-0ee18589648c","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613024135.181851_48.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:15:35.181851Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613024279.610246_53.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:17:59.610246Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:18:14.163 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:18:14.203 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_recreate ... ok 2021-02-11T06:18:14.204 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xscaju-7 2021-02-11T06:18:14.233 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xscaju-8 2021-02-11T06:18:14.288 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xscaju-9 2021-02-11T06:18:14.346 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:18:14.346 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:18:14.346 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:18:14.381 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:18:14.381 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:18:14.608 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=7c4a812f-81e4-4a72-acce-0ee18589648c 2021-02-11T06:18:14.644 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613024135.181851_48.1","last_update":"2021-02-11T06:15:35.181851Z"},{"marker":"1_1613024294.322067_67.1","last_update":"2021-02-11T06:18:14.322067Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:18:14.644 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T06:18:14.645 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:18:14.645 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:18:14.645 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:18:14.686 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:18:14.686 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:18:14.931 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7c4a812f-81e4-4a72-acce-0ee18589648c","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613024135.181851_48.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:15:35.181851Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613024294.313905_65.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:14.313905Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:18:14.932 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1613024294.322067_67.1 target=1_1613024294.313905_65.1 2021-02-11T06:18:19.932 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:18:19.933 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:18:19.933 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:18:19.974 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:18:19.974 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:18:20.220 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7c4a812f-81e4-4a72-acce-0ee18589648c","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613024135.181851_48.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:15:35.181851Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613024294.322067_67.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:14.322067Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:18:20.221 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T06:18:20.221 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T06:18:20.221 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:18:20.221 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:18:20.221 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:18:20.264 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:18:20.264 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:18:20.532 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7c4a812f-81e4-4a72-acce-0ee18589648c","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613024135.181851_48.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:15:35.181851Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613024294.322067_67.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:14.322067Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:18:20.532 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T06:18:20.532 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:18:20.532 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:18:20.532 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:18:20.533 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:18:20.574 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:18:20.574 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:18:20.819 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7c4a812f-81e4-4a72-acce-0ee18589648c","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613024135.181851_48.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:15:35.181851Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613024294.322067_67.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:14.322067Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:18:20.820 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:18:21.198 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:18:23.621 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:18:23.621 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:18:23.664 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:18:23.664 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:18:23.893 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=7c4a812f-81e4-4a72-acce-0ee18589648c 2021-02-11T06:18:23.926 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613024135.181851_48.1","last_update":"2021-02-11T06:15:35.181851Z"},{"marker":"1_1613024301.142235_74.1","last_update":"2021-02-11T06:18:21.142235Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:18:23.926 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T06:18:23.926 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:18:23.927 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:18:23.927 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:18:23.969 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:18:23.970 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:18:24.220 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7c4a812f-81e4-4a72-acce-0ee18589648c","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613024135.181851_48.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:15:35.181851Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:18:24.221 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T06:18:24.221 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T06:18:24.221 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:18:24.221 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:18:24.222 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:18:24.265 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:18:24.265 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:18:24.516 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7c4a812f-81e4-4a72-acce-0ee18589648c","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613024135.181851_48.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:15:35.181851Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:18:24.516 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T06:18:24.517 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:18:24.517 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:18:24.517 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:18:24.517 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:18:24.558 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:18:24.558 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:18:24.805 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7c4a812f-81e4-4a72-acce-0ee18589648c","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613024135.181851_48.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:15:35.181851Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:18:24.806 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:18:24.822 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_remove ... ok 2021-02-11T06:18:24.823 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xscaju-10 2021-02-11T06:18:24.857 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xscaju-11 2021-02-11T06:18:24.916 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xscaju-12 2021-02-11T06:18:25.006 INFO:tasks.rgw.c1.client.1.smithi061.stdout:2021-02-11T06:18:25.002+0000 7ff3aefed700 -1 data sync zone:5f91a5b1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T06:18:25.022 INFO:tasks.rgw.c2.client.0.smithi172.stdout:2021-02-11T06:18:25.015+0000 7fb3be7ec700 -1 data sync zone:5f91a5b1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T06:18:25.023 INFO:tasks.rgw.c2.client.1.smithi172.stdout:2021-02-11T06:18:25.019+0000 7f2bbffef700 -1 data sync zone:5f91a5b1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T06:18:25.583 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:18:25.583 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:18:25.583 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:18:25.644 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:18:25.644 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:18:25.888 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=7c4a812f-81e4-4a72-acce-0ee18589648c 2021-02-11T06:18:25.918 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613024304.951770_72.1","last_update":"2021-02-11T06:18:24.951770Z"},{"marker":"1_1613024301.142235_74.1","last_update":"2021-02-11T06:18:21.142235Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:18:25.918 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T06:18:25.919 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:18:25.919 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:18:25.919 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:18:25.957 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:18:25.957 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:18:26.197 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7c4a812f-81e4-4a72-acce-0ee18589648c","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613024304.951770_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:18:24.951770Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:18:41.974 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T06:18:41.974 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T06:18:41.974 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:18:41.974 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:18:41.975 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:18:42.015 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:18:42.016 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:18:42.262 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7c4a812f-81e4-4a72-acce-0ee18589648c","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613024304.951770_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:18:24.951770Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:18:42.262 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T06:18:42.262 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:18:42.262 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:18:42.262 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:18:42.263 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:18:42.302 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:18:42.302 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:18:42.565 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7c4a812f-81e4-4a72-acce-0ee18589648c","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613024304.951770_72.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:18:24.951770Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:18:42.565 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:18:42.566 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:18:42.566 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:18:42.566 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:18:42.655 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:18:42.655 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:18:42.875 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:18:42.875+0000 7f9ee0fff080 1 bucket sync caught up with source: 2021-02-11T06:18:42.875 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000001.4.5, , 00000000001.3.5, , , , 00000000001.4.5, 00000000001.8.5, , ] 2021-02-11T06:18:42.875 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000001.4.5, , 00000000001.3.5, , , , 00000000001.4.5, 00000000001.8.5, , ] 2021-02-11T06:18:42.876 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:18:42.875+0000 7f9ee0fff080 0 bucket checkpoint complete 2021-02-11T06:18:42.901 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-10 zones={test-zone2, test-zone1} 2021-02-11T06:18:42.912 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:18:42.965 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-02-11T06:18:42.965 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-02-11T06:18:42.965 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-02-11T06:18:42.965 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T06:18:42.965 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:18:42.972 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-02-11T06:18:42.972 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-02-11T06:18:42.973 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-02-11T06:18:42.973 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T06:18:43.030 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-11T06:18:43.084 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-11T06:18:43.204 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-11T06:18:43.310 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-11T06:18:43.364 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-11T06:18:43.479 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-11T06:18:43.586 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-11T06:18:43.640 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-11T06:18:43.756 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-11T06:18:43.862 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T06:18:43.916 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T06:18:44.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T06:18:44.145 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-10 zones={test-zone2, test-zone1} 2021-02-11T06:18:44.206 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:18:44.206 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:18:44.206 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:18:44.253 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:18:44.253 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:18:44.477 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:18:44.474+0000 7f688bf5f080 1 bucket sync caught up with source: 2021-02-11T06:18:44.478 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, 00000000001.4.5, , 00000000001.3.5, , , , 00000000001.4.5, 00000000001.8.5, , ] 2021-02-11T06:18:44.478 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, 00000000001.4.5, , 00000000001.3.5, , , , 00000000001.4.5, 00000000001.8.5, , ] 2021-02-11T06:18:44.478 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:18:44.474+0000 7f688bf5f080 0 bucket checkpoint complete 2021-02-11T06:18:44.505 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-10 zones={test-zone3, test-zone1} 2021-02-11T06:18:44.514 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:18:44.520 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-02-11T06:18:44.520 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-02-11T06:18:44.521 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-02-11T06:18:44.521 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T06:18:44.521 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:18:44.525 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-02-11T06:18:44.525 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-02-11T06:18:44.525 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-02-11T06:18:44.525 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T06:18:44.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-11T06:18:44.588 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-11T06:18:44.699 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-11T06:18:44.801 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-11T06:18:44.856 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-11T06:18:44.971 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-11T06:18:45.070 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-11T06:18:45.122 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-11T06:18:45.237 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-11T06:18:45.340 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T06:18:45.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T06:18:45.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T06:18:45.608 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-10 zones={test-zone3, test-zone1} 2021-02-11T06:18:45.608 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:18:45.609 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:18:45.609 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:18:45.657 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:18:45.657 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:18:45.888 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:18:45.886+0000 7f594383f080 1 bucket sync caught up with source: 2021-02-11T06:18:45.888 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000001.4.5, , 00000000001.3.5, , , , 00000000001.4.5, 00000000001.8.5, , ] 2021-02-11T06:18:45.888 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000001.4.5, , 00000000001.3.5, , , , 00000000001.4.5, 00000000001.8.5, , ] 2021-02-11T06:18:45.889 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:18:45.886+0000 7f594383f080 0 bucket checkpoint complete 2021-02-11T06:18:45.917 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:18:45.917 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:18:45.917 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-11 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:18:45.966 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:18:45.966 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:18:46.196 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:18:46.190+0000 7f8839232080 1 bucket sync caught up with source: 2021-02-11T06:18:46.196 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, 00000000001.11.5, , 00000000001.13.5, , , , 00000000001.14.5, 00000000001.11.5, , ] 2021-02-11T06:18:46.196 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, 00000000001.11.5, , 00000000001.13.5, , , , 00000000001.14.5, 00000000001.11.5, , ] 2021-02-11T06:18:46.196 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:18:46.190+0000 7f8839232080 0 bucket checkpoint complete 2021-02-11T06:18:46.232 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-11 zones={test-zone1, test-zone2} 2021-02-11T06:18:46.242 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:18:46.249 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-02-11T06:18:46.249 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-02-11T06:18:46.249 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-02-11T06:18:46.249 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T06:18:46.249 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:18:46.299 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-02-11T06:18:46.299 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-02-11T06:18:46.299 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-02-11T06:18:46.299 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T06:18:46.314 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-11T06:18:46.378 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-11T06:18:46.514 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-11T06:18:46.628 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-11T06:18:46.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-11T06:18:46.809 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-11T06:18:46.914 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-11T06:18:46.968 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-11T06:18:47.084 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-11T06:18:47.190 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T06:18:47.244 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T06:18:47.354 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T06:18:47.462 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-11 zones={test-zone1, test-zone2} 2021-02-11T06:18:47.463 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:18:47.463 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:18:47.463 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-11 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:18:47.513 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:18:47.513 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:18:47.752 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:18:47.746+0000 7f69cbabe080 1 bucket sync caught up with source: 2021-02-11T06:18:47.752 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, 00000000001.11.5, , 00000000001.13.5, , , , 00000000001.14.5, 00000000001.11.5, , ] 2021-02-11T06:18:47.753 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, 00000000001.11.5, , 00000000001.13.5, , , , 00000000001.14.5, 00000000001.11.5, , ] 2021-02-11T06:18:47.753 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:18:47.746+0000 7f69cbabe080 0 bucket checkpoint complete 2021-02-11T06:18:47.784 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-11 zones={test-zone3, test-zone2} 2021-02-11T06:18:47.795 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:18:47.801 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-02-11T06:18:47.801 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-02-11T06:18:47.801 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-02-11T06:18:47.801 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T06:18:47.801 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:18:47.855 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-02-11T06:18:47.855 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-02-11T06:18:47.855 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-02-11T06:18:47.855 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T06:18:47.911 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-11T06:18:47.969 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-11T06:18:48.082 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-11T06:18:48.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-11T06:18:48.239 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-11T06:18:48.360 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-11T06:18:48.469 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-11T06:18:48.524 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-11T06:18:48.636 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-11T06:18:48.746 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T06:18:48.800 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T06:18:48.911 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T06:18:49.022 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-11 zones={test-zone3, test-zone2} 2021-02-11T06:18:49.022 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:18:49.022 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:18:49.023 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-12 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:18:49.073 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:18:49.073 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:18:49.304 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:18:49.298+0000 7f920ce13080 1 bucket sync caught up with source: 2021-02-11T06:18:49.305 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, 00000000001.12.5, , 00000000001.10.5, , , , 00000000001.10.5, 00000000001.5.5, , ] 2021-02-11T06:18:49.305 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, 00000000001.12.5, , 00000000001.10.5, , , , 00000000001.10.5, 00000000001.5.5, , ] 2021-02-11T06:18:49.305 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:18:49.298+0000 7f920ce13080 0 bucket checkpoint complete 2021-02-11T06:18:49.332 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-12 zones={test-zone1, test-zone3} 2021-02-11T06:18:49.344 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:18:49.351 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-02-11T06:18:49.351 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-02-11T06:18:49.351 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-02-11T06:18:49.351 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T06:18:49.351 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:18:49.355 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-02-11T06:18:49.356 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-02-11T06:18:49.356 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-02-11T06:18:49.356 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T06:18:49.364 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-11T06:18:49.456 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-11T06:18:49.567 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-11T06:18:49.670 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-11T06:18:49.724 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-11T06:18:49.840 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-11T06:18:49.983 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-11T06:18:50.043 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-11T06:18:50.156 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-11T06:18:50.262 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T06:18:50.316 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T06:18:50.427 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T06:18:50.534 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-12 zones={test-zone1, test-zone3} 2021-02-11T06:18:50.534 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:18:50.535 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:18:50.535 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-12 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:18:50.584 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:18:50.584 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:18:50.829 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:18:50.826+0000 7fcefa9c5080 1 bucket sync caught up with source: 2021-02-11T06:18:50.830 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000001.12.5, , 00000000001.10.5, , , , 00000000001.10.5, 00000000001.5.5, , ] 2021-02-11T06:18:50.830 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000001.12.5, , 00000000001.10.5, , , , 00000000001.10.5, 00000000001.5.5, , ] 2021-02-11T06:18:50.830 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:18:50.826+0000 7fcefa9c5080 0 bucket checkpoint complete 2021-02-11T06:18:50.860 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-12 zones={test-zone2, test-zone3} 2021-02-11T06:18:50.871 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:18:50.919 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-02-11T06:18:50.920 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-02-11T06:18:50.920 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-02-11T06:18:50.920 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T06:18:50.920 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:18:50.927 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-02-11T06:18:50.928 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-02-11T06:18:50.928 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-02-11T06:18:50.928 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T06:18:50.986 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-11T06:18:51.041 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-11T06:18:51.151 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-11T06:18:51.254 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-11T06:18:51.308 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-11T06:18:51.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-11T06:19:00.935 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-11T06:19:00.992 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-11T06:19:01.108 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-11T06:19:01.214 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T06:19:01.269 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T06:19:01.379 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T06:19:01.486 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-12 zones={test-zone2, test-zone3} 2021-02-11T06:19:01.487 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_object_sync ... ok 2021-02-11T06:19:01.487 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xscaju-13 2021-02-11T06:19:01.519 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xscaju-14 2021-02-11T06:19:01.580 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xscaju-15 2021-02-11T06:19:01.811 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:19:08.017 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:19:08.018 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:19:08.058 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:19:08.058 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:19:08.294 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=7c4a812f-81e4-4a72-acce-0ee18589648c 2021-02-11T06:19:08.330 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613024341.620645_91.1","last_update":"2021-02-11T06:19:01.620645Z"},{"marker":"1_1613024301.142235_74.1","last_update":"2021-02-11T06:18:21.142235Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:19:08.330 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T06:19:08.331 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:19:08.331 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:19:08.331 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:19:08.371 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:19:08.372 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:19:08.629 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7c4a812f-81e4-4a72-acce-0ee18589648c","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613024341.620645_91.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:19:01.620645Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:19:08.629 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T06:19:08.630 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T06:19:08.630 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:19:08.630 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:19:08.630 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:19:08.674 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:19:08.674 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:19:08.911 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7c4a812f-81e4-4a72-acce-0ee18589648c","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613024341.620645_91.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:19:01.620645Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:19:08.911 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T06:19:08.911 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:19:08.911 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:19:08.912 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:19:08.912 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:19:08.954 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:19:08.954 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:19:09.208 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7c4a812f-81e4-4a72-acce-0ee18589648c","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613024341.620645_91.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:19:01.620645Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:19:09.209 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:19:09.209 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:19:09.209 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:19:09.209 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:19:09.298 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:19:09.298 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:19:09.531 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:19:09.530+0000 7f246112b080 1 bucket sync caught up with source: 2021-02-11T06:19:09.532 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000001.8.5, , , , , , , , , ] 2021-02-11T06:19:09.532 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000001.8.5, , , , , , , , , ] 2021-02-11T06:19:09.532 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:19:09.530+0000 7f246112b080 0 bucket checkpoint complete 2021-02-11T06:19:09.563 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-13 zones={test-zone2, test-zone1} 2021-02-11T06:19:09.575 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:19:09.623 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T06:19:09.623 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:19:09.630 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T06:19:09.685 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T06:19:09.740 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T06:19:09.852 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T06:19:09.959 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-13 zones={test-zone2, test-zone1} 2021-02-11T06:19:09.959 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:19:09.959 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:19:09.959 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:19:10.004 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:19:10.004 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:19:10.216 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:19:10.213+0000 7fe1e95fb080 1 bucket sync caught up with source: 2021-02-11T06:19:10.216 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, 00000000001.8.5, , , , , , , , , ] 2021-02-11T06:19:10.216 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, 00000000001.8.5, , , , , , , , , ] 2021-02-11T06:19:10.216 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:19:10.213+0000 7fe1e95fb080 0 bucket checkpoint complete 2021-02-11T06:19:10.246 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-13 zones={test-zone3, test-zone1} 2021-02-11T06:19:10.256 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:19:10.307 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T06:19:10.307 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:19:10.314 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T06:19:10.373 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T06:19:10.429 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T06:19:10.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T06:19:10.654 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-13 zones={test-zone3, test-zone1} 2021-02-11T06:19:10.654 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:19:10.654 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:19:10.655 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:19:10.702 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:19:10.702 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:19:10.945 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:19:10.941+0000 7fc4fd7d5080 1 bucket sync caught up with source: 2021-02-11T06:19:10.945 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000001.8.5, , , , , , , , , ] 2021-02-11T06:19:10.945 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000001.8.5, , , , , , , , , ] 2021-02-11T06:19:10.945 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:19:10.941+0000 7fc4fd7d5080 0 bucket checkpoint complete 2021-02-11T06:19:10.975 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:19:10.976 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:19:10.976 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:19:11.025 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:19:11.025 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:19:11.264 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:19:11.261+0000 7fd6a3e32080 1 bucket sync caught up with source: 2021-02-11T06:19:11.585 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, 00000000001.8.5, , , , , , , , , ] 2021-02-11T06:19:11.586 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, 00000000001.8.5, , , , , , , , , ] 2021-02-11T06:19:11.586 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:19:11.261+0000 7fd6a3e32080 0 bucket checkpoint complete 2021-02-11T06:19:11.587 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-14 zones={test-zone1, test-zone2} 2021-02-11T06:19:11.602 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:19:11.652 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T06:19:11.652 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:19:11.660 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T06:19:11.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T06:19:11.777 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T06:19:11.887 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T06:19:12.004 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-14 zones={test-zone1, test-zone2} 2021-02-11T06:19:12.004 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:19:12.004 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:19:12.005 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:19:12.053 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:19:12.053 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:19:12.281 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:19:12.277+0000 7f25d6667080 1 bucket sync caught up with source: 2021-02-11T06:19:12.282 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, 00000000001.8.5, , , , , , , , , ] 2021-02-11T06:19:12.282 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, 00000000001.8.5, , , , , , , , , ] 2021-02-11T06:19:12.282 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:19:12.277+0000 7f25d6667080 0 bucket checkpoint complete 2021-02-11T06:19:12.310 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-14 zones={test-zone3, test-zone2} 2021-02-11T06:19:12.319 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:19:12.367 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T06:19:12.367 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:19:12.375 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T06:19:12.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T06:19:12.489 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T06:19:12.604 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T06:19:12.714 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-14 zones={test-zone3, test-zone2} 2021-02-11T06:19:12.714 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:19:12.714 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:19:12.715 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:19:12.762 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:19:12.762 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:19:13.013 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:19:13.009+0000 7f834ca7e080 1 bucket sync caught up with source: 2021-02-11T06:19:13.013 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, 00000000001.10.5, , , , , , , , , ] 2021-02-11T06:19:13.013 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, 00000000001.10.5, , , , , , , , , ] 2021-02-11T06:19:13.013 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:19:13.009+0000 7f834ca7e080 0 bucket checkpoint complete 2021-02-11T06:19:13.049 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-15 zones={test-zone1, test-zone3} 2021-02-11T06:19:13.060 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:19:13.111 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T06:19:13.111 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:19:13.117 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T06:19:13.173 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T06:19:13.228 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T06:19:13.344 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T06:19:13.457 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-15 zones={test-zone1, test-zone3} 2021-02-11T06:19:13.457 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:19:13.458 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:19:13.458 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:19:13.505 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:19:13.506 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:19:13.739 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:19:13.733+0000 7f33107c6080 1 bucket sync caught up with source: 2021-02-11T06:19:13.740 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000001.10.5, , , , , , , , , ] 2021-02-11T06:19:13.740 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000001.10.5, , , , , , , , , ] 2021-02-11T06:19:13.740 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:19:13.733+0000 7f33107c6080 0 bucket checkpoint complete 2021-02-11T06:19:13.769 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-15 zones={test-zone2, test-zone3} 2021-02-11T06:19:13.780 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:19:13.831 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T06:19:13.831 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:19:13.838 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T06:19:13.893 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T06:19:13.948 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T06:19:14.064 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T06:19:14.175 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-15 zones={test-zone2, test-zone3} 2021-02-11T06:19:14.188 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:19:14.188 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:19:14.189 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:19:14.235 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:19:14.235 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:19:14.435 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:19:14.429+0000 7f198628d080 1 bucket sync caught up with source: 2021-02-11T06:19:14.435 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000002.10.5, , , , , , , , , ] 2021-02-11T06:19:14.435 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000002.10.5, , , , , , , , , ] 2021-02-11T06:19:14.435 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:19:14.429+0000 7f198628d080 0 bucket checkpoint complete 2021-02-11T06:19:14.464 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-13 zones={test-zone2, test-zone1} 2021-02-11T06:19:14.475 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:19:14.523 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:19:14.586 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-13 zones={test-zone2, test-zone1} 2021-02-11T06:19:14.586 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:19:14.586 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:19:14.587 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:19:14.632 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:19:14.633 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:19:14.870 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:19:14.869+0000 7f94d5b73080 1 bucket sync caught up with source: 2021-02-11T06:19:14.870 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, 00000000002.10.5, , , , , , , , , ] 2021-02-11T06:19:14.870 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, 00000000002.10.5, , , , , , , , , ] 2021-02-11T06:19:14.871 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:19:14.869+0000 7f94d5b73080 0 bucket checkpoint complete 2021-02-11T06:19:14.897 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-13 zones={test-zone3, test-zone1} 2021-02-11T06:19:14.908 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:19:14.955 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:19:15.020 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-13 zones={test-zone3, test-zone1} 2021-02-11T06:19:15.021 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:19:15.021 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:19:15.021 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:19:15.070 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:19:15.070 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:19:15.307 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:19:15.301+0000 7f815ff10080 1 bucket sync caught up with source: 2021-02-11T06:19:15.307 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000002.10.5, , , , , , , , , ] 2021-02-11T06:19:15.307 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000002.10.5, , , , , , , , , ] 2021-02-11T06:19:15.307 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:19:15.301+0000 7f815ff10080 0 bucket checkpoint complete 2021-02-11T06:19:15.351 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:19:15.352 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:19:15.352 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:19:15.400 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:19:15.401 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:19:15.637 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:19:15.633+0000 7fc73af72080 1 bucket sync caught up with source: 2021-02-11T06:19:15.637 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, 00000000002.10.5, , , , , , , , , ] 2021-02-11T06:19:15.638 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, 00000000002.10.5, , , , , , , , , ] 2021-02-11T06:19:15.638 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:19:15.633+0000 7fc73af72080 0 bucket checkpoint complete 2021-02-11T06:19:15.668 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-14 zones={test-zone1, test-zone2} 2021-02-11T06:19:15.678 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:19:15.726 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:19:15.790 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-14 zones={test-zone1, test-zone2} 2021-02-11T06:19:15.790 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:19:15.790 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:19:15.790 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:19:15.839 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:19:15.839 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:19:16.097 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:19:16.093+0000 7f8aa64b6080 1 bucket sync caught up with source: 2021-02-11T06:19:25.828 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, 00000000002.10.5, , , , , , , , , ] 2021-02-11T06:19:25.828 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, 00000000002.10.5, , , , , , , , , ] 2021-02-11T06:19:25.829 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:19:16.093+0000 7f8aa64b6080 0 bucket checkpoint complete 2021-02-11T06:19:25.830 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-14 zones={test-zone3, test-zone2} 2021-02-11T06:19:25.842 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:19:25.894 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:19:25.964 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-14 zones={test-zone3, test-zone2} 2021-02-11T06:19:32.666 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:19:32.666 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:19:32.667 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:19:32.709 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:19:32.710 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:19:32.929 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:19:32.925+0000 7f4e3539a080 1 bucket sync caught up with source: 2021-02-11T06:19:32.929 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, 00000000002.12.5, , , , , , , , , ] 2021-02-11T06:19:32.929 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, 00000000002.12.5, , , , , , , , , ] 2021-02-11T06:19:32.929 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:19:32.925+0000 7f4e3539a080 0 bucket checkpoint complete 2021-02-11T06:19:32.955 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-15 zones={test-zone1, test-zone3} 2021-02-11T06:19:32.964 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:19:33.011 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:19:33.071 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-15 zones={test-zone1, test-zone3} 2021-02-11T06:19:33.072 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:19:33.072 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:19:33.072 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:19:33.119 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:19:33.119 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:19:33.324 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:19:33.321+0000 7fef25bb3080 1 bucket sync caught up with source: 2021-02-11T06:19:33.324 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000002.12.5, , , , , , , , , ] 2021-02-11T06:19:33.324 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000002.12.5, , , , , , , , , ] 2021-02-11T06:19:33.324 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:19:33.321+0000 7fef25bb3080 0 bucket checkpoint complete 2021-02-11T06:19:33.353 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-15 zones={test-zone2, test-zone3} 2021-02-11T06:19:33.362 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:19:33.411 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:19:33.474 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-15 zones={test-zone2, test-zone3} 2021-02-11T06:19:33.474 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_object_delete ... ok 2021-02-11T06:19:33.475 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xscaju-16 2021-02-11T06:19:33.505 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xscaju-17 2021-02-11T06:19:33.567 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xscaju-18 2021-02-11T06:19:33.701 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:19:33.701 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:19:33.701 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:19:33.734 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:19:33.734 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:19:33.950 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=7c4a812f-81e4-4a72-acce-0ee18589648c 2021-02-11T06:19:33.984 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613024373.689054_115.1","last_update":"2021-02-11T06:19:33.689054Z"},{"marker":"1_1613024301.142235_74.1","last_update":"2021-02-11T06:18:21.142235Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:19:33.985 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T06:19:33.985 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:19:33.985 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:19:33.985 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:19:34.025 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:19:34.025 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:19:34.277 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7c4a812f-81e4-4a72-acce-0ee18589648c","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613024373.689054_115.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:19:33.689054Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:19:34.278 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T06:19:34.278 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T06:19:34.278 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:19:34.278 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:19:34.278 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:19:34.319 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:19:34.320 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:19:34.565 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7c4a812f-81e4-4a72-acce-0ee18589648c","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613024373.689054_115.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:19:33.689054Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:19:34.566 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T06:19:34.566 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:19:34.566 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:19:34.566 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:19:34.566 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:19:34.605 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:19:34.606 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:19:34.888 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7c4a812f-81e4-4a72-acce-0ee18589648c","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613024373.689054_115.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:19:33.689054Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:19:34.888 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:19:34.939 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:19:34.939 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:19:34.939 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:19:34.987 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:19:34.987 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:19:35.225 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:19:35.221+0000 7fd9116ea080 1 bucket sync caught up with source: 2021-02-11T06:19:35.225 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , 00000000002.4.10, , , , , ] 2021-02-11T06:19:35.225 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , 00000000002.4.10, , , , , ] 2021-02-11T06:19:35.225 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:19:35.221+0000 7fd9116ea080 0 bucket checkpoint complete 2021-02-11T06:19:35.252 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:19:35.253 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:19:35.253 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:19:35.299 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:19:35.299 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:19:35.528 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:19:35.525+0000 7fb1a4cbb080 1 bucket sync caught up with source: 2021-02-11T06:19:35.528 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, , , , , 00000000003.6.9, , , , , ] 2021-02-11T06:19:35.528 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, , , , , 00000000003.6.9, , , , , ] 2021-02-11T06:19:35.528 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:19:35.525+0000 7fb1a4cbb080 0 bucket checkpoint complete 2021-02-11T06:19:35.560 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:19:35.560 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:19:35.560 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:19:35.605 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:19:35.605 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:19:35.832 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:19:35.829+0000 7f5e2c05b080 1 bucket sync caught up with source: 2021-02-11T06:19:35.832 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , 00000000005.10.9, , , , , ] 2021-02-11T06:19:35.832 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , 00000000005.10.9, , , , , ] 2021-02-11T06:19:35.832 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:19:35.829+0000 7f5e2c05b080 0 bucket checkpoint complete 2021-02-11T06:19:35.911 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:19:35.911 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:19:35.911 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:19:35.959 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:19:35.959 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:19:36.201 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:19:36.197+0000 7f4382eb3080 1 waiting for incremental sync to catch up: 2021-02-11T06:19:36.876 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, , , , , , , , , , ] 2021-02-11T06:19:36.876 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, , , , , 00000000002.19.10, , , , , ] 2021-02-11T06:19:41.204 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:19:41.201+0000 7f4382eb3080 1 bucket sync caught up with source: 2021-02-11T06:19:53.081 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, , , , , 00000000006.27.9, , , , , ] 2021-02-11T06:19:53.081 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, , , , , 00000000002.19.10, , , , , ] 2021-02-11T06:19:53.081 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:19:41.201+0000 7f4382eb3080 0 bucket checkpoint complete 2021-02-11T06:19:53.083 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:19:53.083 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:19:53.084 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:19:53.129 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:19:53.130 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:19:53.325 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:19:53.321+0000 7f0daddcf080 1 bucket sync caught up with source: 2021-02-11T06:19:53.325 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, , , , , 00000000006.27.9, , , , , ] 2021-02-11T06:19:53.325 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, , , , , 00000000006.27.9, , , , , ] 2021-02-11T06:19:53.325 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:19:53.321+0000 7f0daddcf080 0 bucket checkpoint complete 2021-02-11T06:19:53.407 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:19:53.407 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:19:53.407 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:19:53.457 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:19:53.457 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:19:53.666 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:19:53.665+0000 7fe648db4080 1 bucket sync caught up with source: 2021-02-11T06:19:53.667 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, , , , , 00000000002.14.10, , , , , ] 2021-02-11T06:19:53.667 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, , , , , 00000000002.14.10, , , , , ] 2021-02-11T06:19:53.667 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:19:53.665+0000 7fe648db4080 0 bucket checkpoint complete 2021-02-11T06:19:53.694 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:19:53.694 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:19:53.695 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:19:53.741 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:19:53.741 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:19:53.949 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:19:53.944+0000 7f0d266a0080 1 waiting for incremental sync to catch up: 2021-02-11T06:19:53.950 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , 00000000004.18.9, , , , , ] 2021-02-11T06:19:53.950 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , 00000000006.22.9, , , , , ] 2021-02-11T06:19:58.953 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:19:58.948+0000 7f0d266a0080 1 bucket sync caught up with source: 2021-02-11T06:20:14.247 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , 00000000006.22.9, , , , , ] 2021-02-11T06:20:14.247 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , 00000000006.22.9, , , , , ] 2021-02-11T06:20:14.247 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:19:58.948+0000 7f0d266a0080 0 bucket checkpoint complete 2021-02-11T06:20:15.984 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:20:15.984 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:20:15.984 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:20:16.037 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:20:16.037 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:20:16.235 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:20:16.228+0000 7ff080b8c080 1 bucket sync caught up with source: 2021-02-11T06:20:16.236 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000010.24.5, , 00000000003.17.9, , 00000000008.16.9, , , , , ] 2021-02-11T06:20:16.236 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000010.24.5, , 00000000003.17.9, , 00000000008.16.9, , , , , ] 2021-02-11T06:20:16.236 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:20:16.228+0000 7ff080b8c080 0 bucket checkpoint complete 2021-02-11T06:20:16.265 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:20:16.265 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:20:16.265 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:20:16.316 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:20:16.317 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:20:16.551 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:20:16.548+0000 7f12acfa3080 1 bucket sync caught up with source: 2021-02-11T06:20:16.551 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, 00000000010.24.5, , 00000000003.17.9, , 00000000008.16.9, , , , , ] 2021-02-11T06:20:16.552 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, 00000000010.24.5, , 00000000003.17.9, , 00000000008.16.9, , , , , ] 2021-02-11T06:20:16.552 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:20:16.548+0000 7f12acfa3080 0 bucket checkpoint complete 2021-02-11T06:20:16.579 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:20:16.579 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:20:16.579 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:20:16.629 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:20:16.629 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:20:16.940 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:20:16.935+0000 7fdac1b62080 1 bucket sync caught up with source: 2021-02-11T06:20:40.294 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000010.24.5, , 00000000003.17.9, , 00000000008.16.9, , , , , ] 2021-02-11T06:20:40.294 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000010.24.5, , 00000000003.17.9, , 00000000008.16.9, , , , , ] 2021-02-11T06:20:40.294 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:20:16.935+0000 7fdac1b62080 0 bucket checkpoint complete 2021-02-11T06:20:40.296 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:20:40.296 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:20:40.297 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:20:40.349 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:20:40.349 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:20:40.595 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:20:40.592+0000 7f388b413080 1 bucket sync caught up with source: 2021-02-11T06:20:40.595 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, 00000000003.13.9, , 00000000010.35.5, , 00000000007.16.9, , , , , ] 2021-02-11T06:20:40.595 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, 00000000003.13.9, , 00000000010.35.5, , 00000000007.16.9, , , , , ] 2021-02-11T06:20:40.596 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:20:40.592+0000 7f388b413080 0 bucket checkpoint complete 2021-02-11T06:20:40.632 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:20:40.632 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:20:40.632 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:20:40.721 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:20:40.721 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:20:40.948 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:20:40.944+0000 7fc3cbb07080 1 bucket sync caught up with source: 2021-02-11T06:20:40.948 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, 00000000003.13.9, , 00000000010.35.5, , 00000000007.16.9, , , , , ] 2021-02-11T06:20:40.948 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, 00000000003.13.9, , 00000000010.35.5, , 00000000007.16.9, , , , , ] 2021-02-11T06:20:40.948 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:20:40.944+0000 7fc3cbb07080 0 bucket checkpoint complete 2021-02-11T06:20:40.981 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:20:40.982 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:20:40.982 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:20:41.027 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:20:41.027 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:20:41.259 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:20:41.256+0000 7f124511a080 1 bucket sync caught up with source: 2021-02-11T06:20:41.260 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, 00000000003.15.9, , 00000000003.23.9, , 00000000014.30.5, , , , , ] 2021-02-11T06:20:41.260 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, 00000000003.15.9, , 00000000003.23.9, , 00000000014.30.5, , , , , ] 2021-02-11T06:20:41.260 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:20:41.256+0000 7f124511a080 0 bucket checkpoint complete 2021-02-11T06:20:41.291 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:20:41.291 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:20:41.292 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:20:41.338 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:20:41.338 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:20:41.555 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:20:41.551+0000 7f689885b080 1 bucket sync caught up with source: 2021-02-11T06:20:41.555 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000003.15.9, , 00000000003.23.9, , 00000000014.30.5, , , , , ] 2021-02-11T06:20:41.555 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000003.15.9, , 00000000003.23.9, , 00000000014.30.5, , , , , ] 2021-02-11T06:20:41.556 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:20:41.551+0000 7f689885b080 0 bucket checkpoint complete 2021-02-11T06:20:41.588 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-16 zones={test-zone2, test-zone1} 2021-02-11T06:20:41.599 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:20:41.607 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T06:20:41.607 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T06:20:41.607 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T06:20:41.607 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T06:20:41.607 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:20:41.617 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T06:20:41.618 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T06:20:41.618 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T06:20:41.618 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T06:20:41.636 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T06:20:41.668 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T06:20:41.749 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T06:20:41.813 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T06:20:41.871 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T06:20:41.995 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T06:20:42.100 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T06:20:42.158 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T06:20:42.269 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T06:20:42.374 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T06:20:42.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T06:20:42.545 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T06:20:42.654 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-16 zones={test-zone2, test-zone1} 2021-02-11T06:20:42.654 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-16 zones={test-zone3, test-zone1} 2021-02-11T06:20:42.663 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:20:42.668 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T06:20:42.668 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T06:20:42.668 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T06:20:42.668 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T06:20:42.668 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:20:42.672 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T06:20:42.673 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T06:20:42.673 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T06:20:42.673 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T06:20:42.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T06:20:42.691 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T06:20:42.757 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T06:20:42.817 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T06:20:42.872 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T06:20:42.995 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T06:20:43.098 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T06:20:43.156 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T06:20:43.270 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T06:20:43.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T06:20:43.427 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T06:20:43.538 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T06:20:43.640 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-16 zones={test-zone3, test-zone1} 2021-02-11T06:20:43.640 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-16 zones={test-zone3, test-zone2} 2021-02-11T06:20:43.652 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:20:43.659 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T06:20:43.659 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T06:20:43.659 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T06:20:43.659 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T06:20:43.659 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:20:43.711 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T06:20:43.711 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T06:20:43.711 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T06:20:43.711 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T06:20:43.771 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T06:20:43.784 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T06:20:43.853 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T06:20:43.913 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T06:20:43.968 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T06:20:44.082 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T06:20:44.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T06:20:44.241 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T06:20:44.355 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T06:20:44.461 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T06:20:44.516 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T06:20:44.629 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T06:20:44.733 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-16 zones={test-zone3, test-zone2} 2021-02-11T06:20:44.733 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:20:44.733 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:20:44.734 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:20:44.777 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:20:44.777 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:20:45.008 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:20:45.007+0000 7f3b03483080 1 bucket sync caught up with source: 2021-02-11T06:20:45.008 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000010.33.5, , 00000000003.13.9, , 00000000007.21.9, , , , , ] 2021-02-11T06:20:45.008 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000010.33.5, , 00000000003.13.9, , 00000000007.21.9, , , , , ] 2021-02-11T06:20:45.008 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:20:45.007+0000 7f3b03483080 0 bucket checkpoint complete 2021-02-11T06:20:45.037 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:20:45.037 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:20:45.038 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:20:45.087 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:20:45.087 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:20:45.315 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:20:45.312+0000 7f9191f82080 1 bucket sync caught up with source: 2021-02-11T06:20:45.315 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, 00000000010.33.5, , 00000000003.13.9, , 00000000007.21.9, , , , , ] 2021-02-11T06:20:45.315 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, 00000000010.33.5, , 00000000003.13.9, , 00000000007.21.9, , , , , ] 2021-02-11T06:20:45.315 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:20:45.312+0000 7f9191f82080 0 bucket checkpoint complete 2021-02-11T06:20:45.344 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:20:45.344 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:20:45.344 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:20:45.391 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:20:45.392 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:20:45.607 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:20:45.603+0000 7fbcc5151080 1 bucket sync caught up with source: 2021-02-11T06:20:45.607 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000010.33.5, , 00000000003.13.9, , 00000000007.21.9, , , , , ] 2021-02-11T06:20:45.607 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000010.33.5, , 00000000003.13.9, , 00000000007.21.9, , , , , ] 2021-02-11T06:20:45.607 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:20:45.603+0000 7fbcc5151080 0 bucket checkpoint complete 2021-02-11T06:20:45.637 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:20:45.637 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:20:45.637 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:20:45.686 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:20:45.686 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:20:45.922 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:20:45.920+0000 7f987f926080 1 bucket sync caught up with source: 2021-02-11T06:20:45.922 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, 00000000003.27.9, , 00000000010.33.5, , 00000000009.33.9, , , , , ] 2021-02-11T06:20:45.922 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, 00000000003.27.9, , 00000000010.33.5, , 00000000009.33.9, , , , , ] 2021-02-11T06:20:45.922 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:20:45.920+0000 7f987f926080 0 bucket checkpoint complete 2021-02-11T06:20:45.953 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:20:45.953 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:20:45.953 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:20:45.997 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:20:45.997 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:20:46.244 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:20:46.240+0000 7fdea6c7c080 1 bucket sync caught up with source: 2021-02-11T06:21:06.456 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, 00000000003.27.9, , 00000000010.33.5, , 00000000009.33.9, , , , , ] 2021-02-11T06:21:06.457 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, 00000000003.27.9, , 00000000010.33.5, , 00000000009.33.9, , , , , ] 2021-02-11T06:21:06.457 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:20:46.240+0000 7fdea6c7c080 0 bucket checkpoint complete 2021-02-11T06:21:06.459 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:21:06.459 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:21:06.459 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:21:06.547 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:21:06.547 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:21:06.788 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:21:06.783+0000 7f3d71e28080 1 bucket sync caught up with source: 2021-02-11T06:21:06.788 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, 00000000003.26.9, , 00000000003.16.9, , 00000000014.39.5, , , , , ] 2021-02-11T06:21:06.788 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, 00000000003.26.9, , 00000000003.16.9, , 00000000014.39.5, , , , , ] 2021-02-11T06:21:06.788 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:21:06.783+0000 7f3d71e28080 0 bucket checkpoint complete 2021-02-11T06:21:06.817 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:21:06.817 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:21:06.818 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:21:06.865 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:21:06.865 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:21:07.103 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:21:07.098+0000 7ff7ae0cb080 1 bucket sync caught up with source: 2021-02-11T06:21:07.104 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000003.26.9, , 00000000003.16.9, , 00000000014.39.5, , , , , ] 2021-02-11T06:21:07.104 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000003.26.9, , 00000000003.16.9, , 00000000014.39.5, , , , , ] 2021-02-11T06:21:07.104 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:21:07.098+0000 7ff7ae0cb080 0 bucket checkpoint complete 2021-02-11T06:21:07.133 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-17 zones={test-zone2, test-zone1} 2021-02-11T06:21:07.144 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:21:07.150 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T06:21:07.150 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T06:21:07.150 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T06:21:07.150 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T06:21:07.150 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:21:07.199 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T06:21:07.199 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T06:21:07.199 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T06:21:07.199 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T06:21:07.216 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T06:21:07.225 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T06:21:07.294 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T06:21:07.355 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T06:21:07.409 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T06:21:07.525 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T06:21:07.634 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T06:21:07.688 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T06:21:07.805 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T06:21:07.910 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T06:21:07.964 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T06:21:08.078 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T06:21:08.181 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-17 zones={test-zone2, test-zone1} 2021-02-11T06:21:08.181 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-17 zones={test-zone3, test-zone1} 2021-02-11T06:21:08.192 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:21:08.197 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T06:21:08.198 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T06:21:08.198 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T06:21:08.198 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T06:21:08.198 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:21:08.203 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T06:21:08.203 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T06:21:08.203 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T06:21:08.203 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T06:21:08.212 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T06:21:08.220 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T06:21:08.296 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T06:21:08.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T06:21:08.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T06:21:08.523 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T06:21:08.625 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T06:21:08.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T06:21:08.802 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T06:21:08.906 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T06:21:08.960 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T06:21:09.072 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T06:21:09.172 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-17 zones={test-zone3, test-zone1} 2021-02-11T06:21:09.172 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-17 zones={test-zone3, test-zone2} 2021-02-11T06:21:09.184 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:21:09.189 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T06:21:09.189 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T06:21:09.189 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T06:21:09.190 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T06:21:09.190 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:21:09.239 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T06:21:09.239 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T06:21:09.239 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T06:21:09.239 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T06:21:09.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T06:21:09.306 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T06:21:09.378 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T06:21:09.436 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T06:21:09.493 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T06:21:09.606 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T06:21:09.710 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T06:21:09.765 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T06:21:09.879 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T06:21:09.986 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T06:21:10.040 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T06:21:10.160 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T06:21:10.270 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-17 zones={test-zone3, test-zone2} 2021-02-11T06:21:10.270 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:21:10.270 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:21:10.270 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:21:10.316 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:21:10.316 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:21:10.565 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:21:10.562+0000 7efff9215080 1 bucket sync caught up with source: 2021-02-11T06:21:10.566 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000010.56.5, , 00000000006.25.5, , 00000000007.18.9, , , , , ] 2021-02-11T06:21:10.566 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000010.56.5, , 00000000006.25.5, , 00000000007.18.9, , , , , ] 2021-02-11T06:21:10.566 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:21:10.562+0000 7efff9215080 0 bucket checkpoint complete 2021-02-11T06:21:10.597 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:21:10.597 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:21:10.597 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:21:10.648 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:21:22.914 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:21:22.916 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:21:10.895+0000 7ff4ceed5080 1 bucket sync caught up with source: 2021-02-11T06:21:22.916 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, 00000000010.56.5, , 00000000006.25.5, , 00000000007.18.9, , , , , ] 2021-02-11T06:21:22.916 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, 00000000010.56.5, , 00000000006.25.5, , 00000000007.18.9, , , , , ] 2021-02-11T06:21:22.916 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:21:10.895+0000 7ff4ceed5080 0 bucket checkpoint complete 2021-02-11T06:21:22.917 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:21:22.917 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:21:22.917 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:21:22.965 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:21:22.966 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:21:23.203 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:21:23.197+0000 7f15deeff080 1 bucket sync caught up with source: 2021-02-11T06:21:23.203 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000010.56.5, , 00000000006.25.5, , 00000000007.18.9, , , , , ] 2021-02-11T06:21:23.203 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000010.56.5, , 00000000006.25.5, , 00000000007.18.9, , , , , ] 2021-02-11T06:21:23.203 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:21:23.197+0000 7f15deeff080 0 bucket checkpoint complete 2021-02-11T06:21:23.232 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:21:23.232 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:21:23.233 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:21:23.283 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:21:23.283 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:21:23.526 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:21:23.523+0000 7f4297526080 1 bucket sync caught up with source: 2021-02-11T06:21:23.527 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, 00000000003.33.9, , 00000000010.36.5, , 00000000007.20.9, , , , , ] 2021-02-11T06:21:23.527 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, 00000000003.33.9, , 00000000010.36.5, , 00000000007.20.9, , , , , ] 2021-02-11T06:21:23.527 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:21:23.523+0000 7f4297526080 0 bucket checkpoint complete 2021-02-11T06:21:23.561 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:21:23.561 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:21:23.561 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:21:23.608 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:21:23.608 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:21:23.842 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:21:23.839+0000 7f6d7f855080 1 bucket sync caught up with source: 2021-02-11T06:21:23.843 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, 00000000003.33.9, , 00000000010.36.5, , 00000000007.20.9, , , , , ] 2021-02-11T06:21:23.843 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, 00000000003.33.9, , 00000000010.36.5, , 00000000007.20.9, , , , , ] 2021-02-11T06:21:23.843 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:21:23.839+0000 7f6d7f855080 0 bucket checkpoint complete 2021-02-11T06:21:23.873 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:21:23.873 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:21:23.873 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:21:23.927 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:21:23.928 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:21:24.132 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:21:24.127+0000 7f509c759080 1 bucket sync caught up with source: 2021-02-11T06:21:24.132 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, 00000000003.32.9, , 00000000005.29.5, , 00000000016.42.5, , , , , ] 2021-02-11T06:21:24.132 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, 00000000003.32.9, , 00000000005.29.5, , 00000000016.42.5, , , , , ] 2021-02-11T06:21:24.133 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:21:24.127+0000 7f509c759080 0 bucket checkpoint complete 2021-02-11T06:21:24.165 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:21:24.165 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:21:24.166 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:21:24.215 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:21:24.215 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:21:24.442 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:21:24.437+0000 7ff7e426f080 1 bucket sync caught up with source: 2021-02-11T06:21:24.443 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000003.32.9, , 00000000005.29.5, , 00000000016.42.5, , , , , ] 2021-02-11T06:21:24.443 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000003.32.9, , 00000000005.29.5, , 00000000016.42.5, , , , , ] 2021-02-11T06:21:24.443 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:21:24.437+0000 7ff7e426f080 0 bucket checkpoint complete 2021-02-11T06:21:24.474 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-18 zones={test-zone2, test-zone1} 2021-02-11T06:21:24.484 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:21:24.490 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T06:21:24.490 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T06:21:24.490 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T06:21:24.490 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T06:21:24.490 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:21:24.496 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T06:21:24.496 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T06:21:24.496 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T06:21:24.496 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T06:21:24.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T06:21:24.518 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T06:21:24.584 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T06:21:24.645 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T06:21:24.700 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T06:21:24.816 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T06:21:24.922 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T06:21:24.973 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T06:21:25.091 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T06:21:25.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T06:21:25.251 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T06:21:25.358 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T06:21:25.462 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-18 zones={test-zone2, test-zone1} 2021-02-11T06:21:25.462 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-18 zones={test-zone3, test-zone1} 2021-02-11T06:21:25.473 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:21:25.478 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T06:21:25.478 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T06:21:25.478 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T06:21:25.478 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T06:21:25.478 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:21:25.483 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T06:21:25.483 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T06:21:25.483 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T06:21:25.483 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T06:21:25.493 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T06:21:25.502 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T06:21:25.567 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T06:21:25.631 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T06:21:25.707 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T06:21:26.344 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T06:21:26.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T06:21:26.512 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T06:21:26.634 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T06:21:26.745 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T06:21:26.818 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T06:21:26.942 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T06:21:27.050 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-18 zones={test-zone3, test-zone1} 2021-02-11T06:21:27.051 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-18 zones={test-zone3, test-zone2} 2021-02-11T06:21:27.064 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:21:27.074 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T06:21:27.074 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T06:21:27.074 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T06:21:27.074 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T06:21:27.074 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:21:27.083 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T06:21:27.084 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T06:21:27.084 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T06:21:27.084 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T06:21:27.101 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T06:21:27.117 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T06:21:27.201 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T06:21:27.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T06:21:27.315 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T06:21:27.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T06:21:27.534 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T06:21:27.588 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T06:21:27.708 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T06:21:27.814 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T06:21:27.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T06:21:27.982 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T06:21:28.087 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-18 zones={test-zone3, test-zone2} 2021-02-11T06:21:28.744 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:21:28.744 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:21:28.745 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:21:28.794 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:21:28.794 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:21:28.990 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:21:28.985+0000 7f2c7d69b080 1 bucket sync caught up with source: 2021-02-11T06:21:28.990 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000011.31.5, , 00000000005.24.5, , 00000000009.19.5, , , , , ] 2021-02-11T06:21:28.990 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000011.31.5, , 00000000005.24.5, , 00000000009.19.5, , , , , ] 2021-02-11T06:21:28.990 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:21:28.985+0000 7f2c7d69b080 0 bucket checkpoint complete 2021-02-11T06:21:29.020 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:21:29.021 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:21:29.021 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:21:29.067 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:21:29.067 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:21:29.292 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:21:29.287+0000 7ff8e88f2080 1 bucket sync caught up with source: 2021-02-11T06:21:29.292 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, 00000000011.31.5, , 00000000005.24.5, , 00000000009.19.5, , , , , ] 2021-02-11T06:21:29.293 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, 00000000011.31.5, , 00000000005.24.5, , 00000000009.19.5, , , , , ] 2021-02-11T06:21:29.293 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:21:29.287+0000 7ff8e88f2080 0 bucket checkpoint complete 2021-02-11T06:21:29.319 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:21:29.320 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:21:29.320 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:21:29.363 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:21:29.363 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:21:29.607 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:21:29.601+0000 7fbbcf0c6080 1 bucket sync caught up with source: 2021-02-11T06:21:29.607 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000011.31.5, , 00000000005.24.5, , 00000000009.19.5, , , , , ] 2021-02-11T06:21:29.607 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000011.31.5, , 00000000005.24.5, , 00000000009.19.5, , , , , ] 2021-02-11T06:21:29.608 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:21:29.601+0000 7fbbcf0c6080 0 bucket checkpoint complete 2021-02-11T06:21:29.637 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:21:29.638 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:21:29.638 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:21:29.688 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:21:29.688 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:21:29.919 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:21:29.915+0000 7f774307e080 1 bucket sync caught up with source: 2021-02-11T06:21:29.919 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, 00000000004.16.5, , 00000000011.42.5, , 00000000008.19.5, , , , , ] 2021-02-11T06:21:29.919 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, 00000000004.16.5, , 00000000011.42.5, , 00000000008.19.5, , , , , ] 2021-02-11T06:21:29.919 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:21:29.915+0000 7f774307e080 0 bucket checkpoint complete 2021-02-11T06:21:29.951 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:21:29.951 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:21:29.951 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:21:29.998 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:21:29.998 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:21:30.240 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:21:30.235+0000 7fd935a12080 1 bucket sync caught up with source: 2021-02-11T06:21:30.240 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, 00000000004.16.5, , 00000000011.42.5, , 00000000008.19.5, , , , , ] 2021-02-11T06:21:30.240 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, 00000000004.16.5, , 00000000011.42.5, , 00000000008.19.5, , , , , ] 2021-02-11T06:21:30.241 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:21:30.235+0000 7fd935a12080 0 bucket checkpoint complete 2021-02-11T06:21:30.271 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:21:30.271 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:21:30.272 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:21:30.320 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:21:30.320 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:21:30.572 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:21:30.567+0000 7f558e0bd080 1 bucket sync caught up with source: 2021-02-11T06:21:30.572 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, 00000000004.18.5, , 00000000004.26.5, , 00000000015.33.5, , , , , ] 2021-02-11T06:21:30.572 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, 00000000004.18.5, , 00000000004.26.5, , 00000000015.33.5, , , , , ] 2021-02-11T06:21:30.572 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:21:30.567+0000 7f558e0bd080 0 bucket checkpoint complete 2021-02-11T06:21:30.598 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:21:30.598 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:21:30.599 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:21:30.645 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:21:30.645 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:21:30.882 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:21:30.877+0000 7f6c00edd080 1 bucket sync caught up with source: 2021-02-11T06:21:30.882 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000004.18.5, , 00000000004.26.5, , 00000000015.33.5, , , , , ] 2021-02-11T06:21:30.882 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000004.18.5, , 00000000004.26.5, , 00000000015.33.5, , , , , ] 2021-02-11T06:21:30.882 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:21:30.877+0000 7f6c00edd080 0 bucket checkpoint complete 2021-02-11T06:21:30.913 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-16 zones={test-zone2, test-zone1} 2021-02-11T06:21:30.923 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:21:30.929 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T06:21:30.929 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T06:21:30.929 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T06:21:30.929 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T06:21:30.930 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:21:30.979 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T06:21:30.979 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T06:21:30.979 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T06:21:30.979 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T06:21:30.993 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T06:21:31.004 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T06:21:31.072 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T06:21:31.134 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T06:21:31.188 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T06:21:31.305 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T06:21:31.412 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T06:21:31.468 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T06:21:31.586 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T06:21:31.698 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T06:21:31.752 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T06:21:31.866 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T06:21:31.969 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-16 zones={test-zone2, test-zone1} 2021-02-11T06:21:31.970 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-16 zones={test-zone3, test-zone1} 2021-02-11T06:21:31.980 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:21:31.985 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T06:21:31.985 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T06:21:31.985 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T06:21:31.985 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T06:21:31.985 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:21:31.989 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T06:21:31.989 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T06:21:31.989 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T06:21:31.989 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T06:21:31.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T06:21:32.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T06:21:32.072 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T06:21:32.133 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T06:21:32.192 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T06:21:32.308 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T06:21:32.410 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T06:21:32.464 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T06:21:32.577 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T06:21:32.681 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T06:21:32.736 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T06:21:32.853 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T06:21:38.260 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-16 zones={test-zone3, test-zone1} 2021-02-11T06:21:38.260 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-16 zones={test-zone3, test-zone2} 2021-02-11T06:21:38.279 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:21:38.294 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T06:21:38.294 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T06:21:38.295 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T06:21:38.295 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T06:21:38.295 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:21:38.304 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T06:21:38.304 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T06:21:38.304 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T06:21:38.305 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T06:21:38.320 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T06:21:38.337 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T06:21:38.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T06:21:38.486 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T06:21:38.547 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T06:21:38.667 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T06:21:38.774 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T06:21:38.827 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T06:21:38.940 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T06:21:39.045 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T06:21:39.099 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T06:21:39.210 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T06:21:39.313 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-16 zones={test-zone3, test-zone2} 2021-02-11T06:21:39.314 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:21:39.314 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:21:39.314 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:21:39.361 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:21:39.361 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:21:39.599 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:21:39.593+0000 7fd96d1bc080 1 bucket sync caught up with source: 2021-02-11T06:21:39.599 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000011.62.5, , 00000000004.16.5, , 00000000008.24.5, , , , , ] 2021-02-11T06:21:39.599 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000011.62.5, , 00000000004.16.5, , 00000000008.24.5, , , , , ] 2021-02-11T06:21:39.600 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:21:39.593+0000 7fd96d1bc080 0 bucket checkpoint complete 2021-02-11T06:21:39.630 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:21:39.630 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:21:39.630 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:21:39.679 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:21:39.679 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:21:39.914 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:21:39.911+0000 7f12b507e080 1 bucket sync caught up with source: 2021-02-11T06:21:39.914 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, 00000000011.62.5, , 00000000004.16.5, , 00000000008.24.5, , , , , ] 2021-02-11T06:21:39.914 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, 00000000011.62.5, , 00000000004.16.5, , 00000000008.24.5, , , , , ] 2021-02-11T06:21:39.914 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:21:39.911+0000 7f12b507e080 0 bucket checkpoint complete 2021-02-11T06:21:39.942 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:21:39.943 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:21:39.943 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:21:39.989 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:21:39.989 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:21:40.227 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:21:40.221+0000 7f973afd8080 1 bucket sync caught up with source: 2021-02-11T06:21:40.228 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000011.62.5, , 00000000004.16.5, , 00000000008.24.5, , , , , ] 2021-02-11T06:21:40.228 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000011.62.5, , 00000000004.16.5, , 00000000008.24.5, , , , , ] 2021-02-11T06:21:40.228 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:21:40.221+0000 7f973afd8080 0 bucket checkpoint complete 2021-02-11T06:21:40.252 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:21:40.252 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:21:40.252 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:21:40.299 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:21:40.299 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:21:40.555 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:21:40.547+0000 7fa23e78e080 1 bucket sync caught up with source: 2021-02-11T06:21:40.555 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, 00000000004.36.5, , 00000000011.40.5, , 00000000010.36.5, , , , , ] 2021-02-11T06:21:40.555 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, 00000000004.36.5, , 00000000011.40.5, , 00000000010.36.5, , , , , ] 2021-02-11T06:21:40.555 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:21:40.547+0000 7fa23e78e080 0 bucket checkpoint complete 2021-02-11T06:21:40.587 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:21:40.587 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:21:40.587 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:21:40.635 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:21:40.635 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:21:40.897 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:21:40.891+0000 7f0459fe1080 1 bucket sync caught up with source: 2021-02-11T06:21:40.897 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, 00000000004.36.5, , 00000000011.40.5, , 00000000010.36.5, , , , , ] 2021-02-11T06:21:40.897 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, 00000000004.36.5, , 00000000011.40.5, , 00000000010.36.5, , , , , ] 2021-02-11T06:21:40.897 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:21:40.891+0000 7f0459fe1080 0 bucket checkpoint complete 2021-02-11T06:21:40.926 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:21:40.926 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:21:40.927 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:21:40.972 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:21:40.972 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:21:41.218 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:21:41.215+0000 7f2ed1006080 1 bucket sync caught up with source: 2021-02-11T06:21:41.219 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, 00000000004.35.5, , 00000000004.19.5, , 00000000015.42.5, , , , , ] 2021-02-11T06:21:41.219 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, 00000000004.35.5, , 00000000004.19.5, , 00000000015.42.5, , , , , ] 2021-02-11T06:21:41.219 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:21:41.215+0000 7f2ed1006080 0 bucket checkpoint complete 2021-02-11T06:21:41.246 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:21:41.246 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:21:41.246 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:21:41.294 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:21:41.294 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:21:41.523 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:21:41.517+0000 7f6eb87b0080 1 bucket sync caught up with source: 2021-02-11T06:21:41.523 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000004.35.5, , 00000000004.19.5, , 00000000015.42.5, , , , , ] 2021-02-11T06:21:41.523 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000004.35.5, , 00000000004.19.5, , 00000000015.42.5, , , , , ] 2021-02-11T06:21:41.523 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:21:41.517+0000 7f6eb87b0080 0 bucket checkpoint complete 2021-02-11T06:21:41.554 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-17 zones={test-zone2, test-zone1} 2021-02-11T06:21:41.564 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:21:41.611 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T06:21:41.611 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T06:21:41.611 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T06:21:41.611 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T06:21:41.611 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:21:41.619 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T06:21:41.619 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T06:21:41.619 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T06:21:41.619 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T06:21:41.675 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T06:21:41.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T06:21:41.753 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T06:21:41.817 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T06:21:41.873 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T06:21:41.987 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T06:21:42.094 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T06:21:42.148 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T06:21:42.265 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T06:21:42.370 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T06:21:42.423 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T06:21:42.537 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T06:21:42.641 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-17 zones={test-zone2, test-zone1} 2021-02-11T06:21:42.642 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-17 zones={test-zone3, test-zone1} 2021-02-11T06:21:42.652 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:21:42.657 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T06:21:42.657 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T06:21:42.657 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T06:21:42.657 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T06:21:42.657 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:21:42.662 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T06:21:42.662 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T06:21:42.662 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T06:21:42.662 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T06:21:42.671 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T06:21:42.679 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T06:21:42.745 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T06:21:42.801 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T06:21:42.855 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T06:21:42.974 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T06:21:43.077 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T06:21:43.137 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T06:21:43.249 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T06:21:43.352 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T06:21:43.408 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T06:21:43.521 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T06:21:43.625 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-17 zones={test-zone3, test-zone1} 2021-02-11T06:21:43.626 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-17 zones={test-zone3, test-zone2} 2021-02-11T06:21:43.638 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:21:43.643 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T06:21:43.643 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T06:21:43.643 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T06:21:43.643 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T06:21:43.644 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:21:43.650 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T06:21:43.650 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T06:21:43.650 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T06:21:43.650 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T06:21:43.710 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T06:21:43.722 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T06:21:43.790 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T06:21:43.853 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T06:21:43.908 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T06:21:44.021 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T06:21:44.126 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T06:21:44.180 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T06:21:44.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T06:21:44.401 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T06:21:44.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T06:21:44.568 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T06:21:44.675 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-17 zones={test-zone3, test-zone2} 2021-02-11T06:21:44.675 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:21:44.675 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:21:44.675 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:21:44.718 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:21:44.718 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:21:44.947 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:21:44.941+0000 7fbf11aa2080 1 bucket sync caught up with source: 2021-02-11T06:21:46.252 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000011.64.5, , 00000000007.28.5, , 00000000008.21.5, , , , , ] 2021-02-11T06:21:46.253 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000011.64.5, , 00000000007.28.5, , 00000000008.21.5, , , , , ] 2021-02-11T06:21:46.253 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:21:44.941+0000 7fbf11aa2080 0 bucket checkpoint complete 2021-02-11T06:21:46.254 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:21:46.254 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:21:46.254 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:21:46.303 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:21:46.303 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:21:46.530 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:21:46.527+0000 7f2d3d91a080 1 bucket sync caught up with source: 2021-02-11T06:21:46.530 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, 00000000011.64.5, , 00000000007.28.5, , 00000000008.21.5, , , , , ] 2021-02-11T06:21:46.530 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, 00000000011.64.5, , 00000000007.28.5, , 00000000008.21.5, , , , , ] 2021-02-11T06:21:46.530 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:21:46.527+0000 7f2d3d91a080 0 bucket checkpoint complete 2021-02-11T06:21:46.559 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:21:46.559 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:21:46.560 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:21:46.607 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:21:46.607 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:21:46.845 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:21:46.841+0000 7fe2f831d080 1 bucket sync caught up with source: 2021-02-11T06:21:46.845 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000011.64.5, , 00000000007.28.5, , 00000000008.21.5, , , , , ] 2021-02-11T06:21:46.845 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000011.64.5, , 00000000007.28.5, , 00000000008.21.5, , , , , ] 2021-02-11T06:21:46.845 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:21:46.841+0000 7fe2f831d080 0 bucket checkpoint complete 2021-02-11T06:21:46.875 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:21:46.875 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:21:46.876 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:21:46.923 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:21:46.923 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:21:47.161 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:21:47.155+0000 7fb6182ec080 1 bucket sync caught up with source: 2021-02-11T06:21:47.161 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, 00000000004.38.5, , 00000000011.42.5, , 00000000008.23.5, , , , , ] 2021-02-11T06:21:47.161 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, 00000000004.38.5, , 00000000011.42.5, , 00000000008.23.5, , , , , ] 2021-02-11T06:21:47.161 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:21:47.155+0000 7fb6182ec080 0 bucket checkpoint complete 2021-02-11T06:21:47.192 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:21:47.192 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:21:47.192 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:21:47.237 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:21:47.237 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:21:47.459 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:21:47.455+0000 7f67f5516080 1 bucket sync caught up with source: 2021-02-11T06:21:47.459 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, 00000000004.38.5, , 00000000011.42.5, , 00000000008.23.5, , , , , ] 2021-02-11T06:21:47.459 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, 00000000004.38.5, , 00000000011.42.5, , 00000000008.23.5, , , , , ] 2021-02-11T06:21:47.460 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:21:47.455+0000 7f67f5516080 0 bucket checkpoint complete 2021-02-11T06:21:47.492 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:21:47.492 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:21:47.492 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:21:47.580 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:21:47.581 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:21:47.826 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:21:47.823+0000 7f529339b080 1 bucket sync caught up with source: 2021-02-11T06:21:47.826 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, 00000000004.37.5, , 00000000006.34.5, , 00000000017.45.5, , , , , ] 2021-02-11T06:21:47.826 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, 00000000004.37.5, , 00000000006.34.5, , 00000000017.45.5, , , , , ] 2021-02-11T06:21:47.826 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:21:47.823+0000 7f529339b080 0 bucket checkpoint complete 2021-02-11T06:21:47.855 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:21:47.855 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:21:47.855 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:21:47.901 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:21:47.901 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:21:48.123 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:21:48.117+0000 7ff00213b080 1 bucket sync caught up with source: 2021-02-11T06:21:48.123 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000004.37.5, , 00000000006.34.5, , 00000000017.45.5, , , , , ] 2021-02-11T06:21:48.123 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000004.37.5, , 00000000006.34.5, , 00000000017.45.5, , , , , ] 2021-02-11T06:21:48.123 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:21:48.117+0000 7ff00213b080 0 bucket checkpoint complete 2021-02-11T06:21:48.152 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-18 zones={test-zone2, test-zone1} 2021-02-11T06:21:48.163 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:21:48.168 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T06:21:48.168 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T06:21:48.168 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T06:21:48.168 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T06:21:48.169 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:21:48.174 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T06:21:48.175 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T06:21:48.175 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T06:21:48.175 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T06:21:48.230 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T06:21:48.241 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T06:21:48.314 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T06:21:48.373 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T06:21:48.428 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T06:21:48.574 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T06:21:48.689 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T06:21:48.749 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T06:21:48.873 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T06:21:48.987 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T06:21:49.045 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T06:21:49.156 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T06:21:49.266 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-18 zones={test-zone2, test-zone1} 2021-02-11T06:21:49.266 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-18 zones={test-zone3, test-zone1} 2021-02-11T06:21:49.275 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:21:49.280 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T06:21:49.280 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T06:21:49.280 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T06:21:49.280 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T06:21:49.280 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:21:49.285 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T06:21:49.285 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T06:21:49.285 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T06:21:49.285 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T06:21:49.294 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T06:21:49.303 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T06:21:49.368 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T06:21:49.425 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T06:21:49.480 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T06:21:49.593 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T06:21:49.698 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T06:21:49.753 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T06:21:49.869 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T06:21:49.973 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T06:21:50.028 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T06:21:50.141 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T06:21:50.245 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-18 zones={test-zone3, test-zone1} 2021-02-11T06:21:50.245 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-18 zones={test-zone3, test-zone2} 2021-02-11T06:21:50.258 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:21:50.262 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T06:21:50.263 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T06:21:50.263 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T06:21:50.263 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T06:21:50.263 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:21:50.268 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T06:21:50.268 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T06:21:50.268 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T06:21:50.268 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T06:21:50.278 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T06:21:50.286 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T06:21:50.353 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T06:21:50.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T06:21:50.468 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T06:21:50.581 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T06:21:50.686 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T06:21:50.740 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T06:21:50.859 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T06:21:50.966 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T06:21:51.020 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T06:21:51.137 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T06:21:51.242 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-18 zones={test-zone3, test-zone2} 2021-02-11T06:21:51.243 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_versioned_object_incremental_sync ... ok 2021-02-11T06:21:51.289 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:21:51.289 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:21:51.289 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:21:51.334 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:21:51.334 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:21:51.542 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=7c4a812f-81e4-4a72-acce-0ee18589648c 2021-02-11T06:21:51.578 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613024511.287525_138.1","last_update":"2021-02-11T06:21:51.287525Z"},{"marker":"1_1613024301.142235_74.1","last_update":"2021-02-11T06:18:21.142235Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:21:51.578 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T06:21:51.578 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:21:51.578 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:21:51.579 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:21:51.618 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:21:51.618 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:21:51.860 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7c4a812f-81e4-4a72-acce-0ee18589648c","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613024511.287525_138.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:21:51.287525Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:21:51.861 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T06:21:51.861 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T06:21:51.861 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:21:51.861 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:21:51.861 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:21:51.903 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:21:51.903 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:21:52.166 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7c4a812f-81e4-4a72-acce-0ee18589648c","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613024511.287525_138.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:21:51.287525Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:21:52.166 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T06:21:52.166 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:21:52.167 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:21:52.167 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:21:52.167 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:21:52.208 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:21:52.208 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:21:52.450 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7c4a812f-81e4-4a72-acce-0ee18589648c","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613024511.287525_138.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:21:51.287525Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:21:52.450 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:21:52.503 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:21:52.503 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:21:52.503 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:21:52.553 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:21:52.553 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:21:52.794 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:21:52.789+0000 7f1a03309080 1 waiting for incremental sync to catch up: 2021-02-11T06:21:52.795 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , 00000000002.11.10, , , , , ] 2021-02-11T06:21:52.795 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , 00000000003.13.9, , , , , ] 2021-02-11T06:21:57.798 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:21:57.792+0000 7f1a03309080 1 bucket sync caught up with source: 2021-02-11T06:22:07.158 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , 00000000005.17.9, , , , , ] 2021-02-11T06:22:07.158 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , 00000000003.13.9, , , , , ] 2021-02-11T06:22:07.158 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:21:57.792+0000 7f1a03309080 0 bucket checkpoint complete 2021-02-11T06:22:07.159 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:22:07.160 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:22:07.160 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:22:07.206 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:22:07.207 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:22:07.436 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:22:07.431+0000 7fef9bd3c080 1 bucket sync caught up with source: 2021-02-11T06:22:07.437 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, , , , , 00000000005.17.9, , , , , ] 2021-02-11T06:22:07.437 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, , , , , 00000000005.17.9, , , , , ] 2021-02-11T06:22:07.437 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:22:07.431+0000 7fef9bd3c080 0 bucket checkpoint complete 2021-02-11T06:22:07.462 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:22:07.462 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:22:07.463 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:22:07.509 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:22:07.510 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:22:07.743 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:22:07.736+0000 7f0a330d9080 1 bucket sync caught up with source: 2021-02-11T06:22:07.743 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , 00000000005.17.9, , , , , ] 2021-02-11T06:22:07.743 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , 00000000005.17.9, , , , , ] 2021-02-11T06:22:07.743 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:22:07.736+0000 7f0a330d9080 0 bucket checkpoint complete 2021-02-11T06:22:07.773 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:22:07.773 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:22:07.774 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:22:07.823 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:22:07.824 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:22:08.054 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:22:08.051+0000 7fef371af080 1 bucket sync caught up with source: 2021-02-11T06:22:08.054 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, , , , , 00000000005.25.9, , , , , ] 2021-02-11T06:22:08.055 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, , , , , 00000000005.25.9, , , , , ] 2021-02-11T06:22:08.055 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:22:08.051+0000 7fef371af080 0 bucket checkpoint complete 2021-02-11T06:22:08.088 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:22:08.089 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:22:08.089 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:22:08.135 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:22:08.136 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:22:08.366 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:22:08.363+0000 7fba64c26080 1 bucket sync caught up with source: 2021-02-11T06:22:08.366 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, , , , , 00000000005.25.9, , , , , ] 2021-02-11T06:22:08.366 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, , , , , 00000000005.25.9, , , , , ] 2021-02-11T06:22:08.366 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:22:08.363+0000 7fba64c26080 0 bucket checkpoint complete 2021-02-11T06:22:08.399 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:22:08.399 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:22:08.399 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:22:08.448 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:22:08.448 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:22:08.703 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:22:08.699+0000 7f2759403080 1 bucket sync caught up with source: 2021-02-11T06:22:25.916 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, , , , , 00000000004.19.9, , , , , ] 2021-02-11T06:22:25.916 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, , , , , 00000000004.19.9, , , , , ] 2021-02-11T06:22:25.916 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:22:08.699+0000 7f2759403080 0 bucket checkpoint complete 2021-02-11T06:22:25.918 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:22:25.918 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:22:25.919 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:22:25.967 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:22:25.967 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:22:26.197 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:22:26.192+0000 7fbb77914080 1 bucket sync caught up with source: 2021-02-11T06:22:26.197 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , 00000000004.19.9, , , , , ] 2021-02-11T06:22:26.197 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , 00000000004.19.9, , , , , ] 2021-02-11T06:22:26.197 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:22:26.192+0000 7fbb77914080 0 bucket checkpoint complete 2021-02-11T06:22:26.228 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-19 zones={test-zone2, test-zone1} 2021-02-11T06:22:26.238 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:22:26.288 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T06:22:26.288 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:22:26.295 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T06:22:26.349 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T06:22:26.360 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T06:22:26.429 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T06:22:26.489 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-19 zones={test-zone2, test-zone1} 2021-02-11T06:22:26.489 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-19 zones={test-zone3, test-zone1} 2021-02-11T06:22:26.497 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:22:26.543 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T06:22:26.543 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:22:26.550 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T06:22:26.606 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T06:22:26.617 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T06:22:26.680 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T06:22:26.743 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-19 zones={test-zone3, test-zone1} 2021-02-11T06:22:30.056 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-19 zones={test-zone3, test-zone2} 2021-02-11T06:22:30.068 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:22:30.115 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T06:22:30.115 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:22:30.122 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T06:22:30.177 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T06:22:30.188 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T06:22:30.254 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T06:22:30.313 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-19 zones={test-zone3, test-zone2} 2021-02-11T06:22:31.851 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:22:31.851 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:22:31.851 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:22:31.900 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:22:31.900 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:22:32.114 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:22:32.107+0000 7fa34b3b5080 1 waiting for incremental sync to catch up: 2021-02-11T06:22:32.115 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , 00000000005.17.9, , , 00000000049.133.5, , ] 2021-02-11T06:22:32.115 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , 00000000005.17.9, , , 00000000051.138.8, , ] 2021-02-11T06:22:37.120 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:22:37.111+0000 7fa34b3b5080 1 bucket sync caught up with source: 2021-02-11T06:22:37.120 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , 00000000005.17.9, , , 00000000055.148.9, , ] 2021-02-11T06:22:37.120 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , 00000000005.17.9, , , 00000000051.138.8, , ] 2021-02-11T06:22:37.120 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:22:37.111+0000 7fa34b3b5080 0 bucket checkpoint complete 2021-02-11T06:22:37.152 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:22:37.152 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:22:37.153 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:22:37.200 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:22:37.200 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:22:37.438 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:22:37.434+0000 7f45d694b080 1 bucket sync caught up with source: 2021-02-11T06:22:37.438 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, , , , , 00000000005.17.9, , , 00000000055.148.9, , ] 2021-02-11T06:22:37.438 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, , , , , 00000000005.17.9, , , 00000000055.148.9, , ] 2021-02-11T06:22:37.438 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:22:37.434+0000 7f45d694b080 0 bucket checkpoint complete 2021-02-11T06:22:37.464 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:22:37.464 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:22:37.465 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:22:37.513 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:22:37.514 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:22:37.750 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:22:37.743+0000 7f953d4d5080 1 bucket sync caught up with source: 2021-02-11T06:22:37.750 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , 00000000005.17.9, , , 00000000055.148.9, , ] 2021-02-11T06:22:37.750 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , 00000000005.17.9, , , 00000000055.148.9, , ] 2021-02-11T06:22:37.750 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:22:37.743+0000 7f953d4d5080 0 bucket checkpoint complete 2021-02-11T06:22:37.779 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:22:37.780 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:22:37.780 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:22:37.830 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:22:37.830 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:22:38.083 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:22:38.078+0000 7fdda9d8f080 1 bucket sync caught up with source: 2021-02-11T06:22:56.368 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, , , , , 00000000005.25.9, , , 00000000052.131.9, , ] 2021-02-11T06:22:56.368 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, , , , , 00000000005.25.9, , , 00000000052.131.9, , ] 2021-02-11T06:22:56.368 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:22:38.078+0000 7fdda9d8f080 0 bucket checkpoint complete 2021-02-11T06:22:56.370 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:22:56.370 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:22:56.371 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:22:56.419 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:22:56.420 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:22:56.668 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:22:56.662+0000 7f0c424bb080 1 bucket sync caught up with source: 2021-02-11T06:22:56.668 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, , , , , 00000000005.25.9, , , 00000000052.131.9, , ] 2021-02-11T06:22:56.668 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, , , , , 00000000005.25.9, , , 00000000052.131.9, , ] 2021-02-11T06:22:56.668 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:22:56.662+0000 7f0c424bb080 0 bucket checkpoint complete 2021-02-11T06:22:56.700 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:22:56.700 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:22:56.700 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:22:56.744 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:22:56.745 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:22:57.000 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:22:56.994+0000 7fca77bf6080 1 bucket sync caught up with source: 2021-02-11T06:22:57.000 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, , , , , 00000000004.19.9, , , 00000000059.149.9, , ] 2021-02-11T06:22:57.000 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, , , , , 00000000004.19.9, , , 00000000059.149.9, , ] 2021-02-11T06:22:57.000 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:22:56.994+0000 7fca77bf6080 0 bucket checkpoint complete 2021-02-11T06:22:57.033 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:22:57.033 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:22:57.033 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:22:57.081 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:22:57.081 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:22:57.314 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:22:57.311+0000 7f16da235080 1 bucket sync caught up with source: 2021-02-11T06:22:57.314 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , 00000000004.19.9, , , 00000000059.149.9, , ] 2021-02-11T06:22:57.314 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , 00000000004.19.9, , , 00000000059.149.9, , ] 2021-02-11T06:22:57.314 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:22:57.311+0000 7f16da235080 0 bucket checkpoint complete 2021-02-11T06:22:57.346 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-19 zones={test-zone2, test-zone1} 2021-02-11T06:22:57.357 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:22:57.366 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T06:22:57.366 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:22:57.366 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:22:57.366 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:22:57.367 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:22:57.367 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:22:57.367 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:22:57.367 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:22:57.367 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:22:57.367 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:22:57.367 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:22:57.367 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:22:57.367 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:22:57.367 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:22:57.367 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:22:57.367 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:22:57.367 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:22:57.367 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:22:57.368 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:22:57.368 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:22:57.368 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:22:57.368 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:22:57.368 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:22:57.368 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:22:57.368 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:22:57.368 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:22:57.368 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:22:57.368 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:22:57.368 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:22:57.368 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:22:57.368 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:22:57.369 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:22:57.378 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T06:22:57.379 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:22:57.379 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:22:57.379 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:22:57.379 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:22:57.379 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:22:57.379 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:22:57.379 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:22:57.379 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:22:57.379 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:22:57.379 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:22:57.379 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:22:57.379 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:22:57.379 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:22:57.380 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:22:57.380 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:22:57.380 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:22:57.380 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:22:57.380 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:22:57.380 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:22:57.380 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:22:57.380 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:22:57.380 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:22:57.380 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:22:57.380 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:22:57.380 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:22:57.381 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:22:57.381 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:22:57.381 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:22:57.381 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:22:57.381 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:22:57.401 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T06:22:57.412 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T06:22:57.477 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T06:22:57.536 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:22:57.595 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:22:57.713 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:22:57.820 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:22:57.877 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:22:58.000 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:22:58.112 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:22:58.168 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:22:58.289 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:22:58.396 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:22:58.457 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:22:58.576 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:22:58.684 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:22:58.740 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:22:58.860 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:22:58.969 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:22:59.022 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:22:59.137 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:22:59.247 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:22:59.304 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:22:59.424 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:22:59.539 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:22:59.599 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:22:59.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:22:59.828 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:22:59.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:00.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:00.117 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:00.171 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:00.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:00.403 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:00.467 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:00.630 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:00.736 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:00.797 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:00.916 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:01.027 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:01.080 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:01.205 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:01.310 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:01.363 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:01.488 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:01.600 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:01.656 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:01.777 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:01.888 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:01.944 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:02.065 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:02.171 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:02.228 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:02.348 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:02.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:02.547 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:02.910 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:03.019 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:03.072 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:03.198 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:03.303 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:03.354 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:03.478 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:03.583 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:03.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:03.752 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:03.863 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:03.915 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:04.040 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:04.146 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:04.220 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:04.368 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:04.482 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:04.544 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:04.676 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:04.793 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:04.855 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:04.980 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:05.092 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:05.149 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:05.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:05.368 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:05.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:05.540 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:05.651 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:05.703 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:05.827 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:05.936 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:05.991 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:06.113 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:06.224 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:06.279 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:06.396 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:06.507 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-19 zones={test-zone2, test-zone1} 2021-02-11T06:23:06.507 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-19 zones={test-zone3, test-zone1} 2021-02-11T06:23:06.516 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:23:06.524 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T06:23:06.524 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:06.524 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:06.524 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:06.524 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:06.525 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:06.525 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:06.525 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:06.525 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:06.525 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:06.525 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:06.525 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:06.525 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:06.525 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:06.525 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:06.525 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:06.525 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:06.525 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:06.525 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:06.526 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:06.526 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:06.526 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:06.526 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:06.526 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:06.526 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:06.526 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:06.526 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:06.526 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:06.526 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:06.526 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:06.526 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:06.526 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:23:06.533 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T06:23:06.533 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:06.533 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:06.533 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:06.533 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:06.533 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:06.533 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:06.534 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:06.534 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:06.534 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:06.534 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:06.534 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:06.534 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:06.534 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:06.534 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:06.534 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:06.534 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:06.534 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:06.534 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:06.534 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:06.535 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:06.535 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:06.535 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:06.535 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:06.535 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:06.535 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:06.535 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:06.535 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:06.535 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:06.535 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:06.535 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:06.552 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T06:23:06.561 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T06:23:06.628 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T06:23:06.689 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:06.744 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:06.865 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:06.978 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:07.032 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:07.152 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:07.258 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:07.311 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:07.433 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:07.538 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:07.592 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:07.710 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:07.814 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:07.869 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:07.986 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:08.091 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:08.148 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:08.265 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:08.374 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:08.428 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:08.547 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:08.651 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:08.708 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:08.830 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:08.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:08.984 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:09.104 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:09.211 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:09.264 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:09.381 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:09.486 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:09.540 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:09.658 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:09.764 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:09.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:09.938 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:10.043 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:10.096 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:10.217 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:10.322 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:10.376 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:10.495 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:10.598 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:10.651 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:10.769 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:10.874 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:10.928 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:11.047 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:11.156 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:11.211 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:11.339 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:17.085 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:17.141 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:17.261 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:17.365 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:17.420 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:17.537 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:17.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:17.699 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:17.818 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:17.926 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:17.984 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:18.103 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:18.251 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:18.304 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:18.423 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:18.527 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:18.588 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:18.707 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:18.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:18.868 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:18.987 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:19.090 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:19.145 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:19.265 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:19.370 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:19.425 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:19.546 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:19.652 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:19.707 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:19.821 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:19.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:19.988 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:20.107 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:20.212 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:20.268 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:20.386 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:20.497 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:21.472 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:21.591 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:21.700 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-19 zones={test-zone3, test-zone1} 2021-02-11T06:23:21.700 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-19 zones={test-zone3, test-zone2} 2021-02-11T06:23:21.710 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:23:21.717 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T06:23:21.717 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:21.717 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:21.717 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:21.717 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:21.717 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:21.717 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:21.717 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:21.717 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:21.717 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:21.718 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:21.718 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:21.718 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:21.718 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:21.718 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:21.718 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:21.718 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:21.718 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:21.718 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:21.718 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:21.718 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:21.718 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:21.718 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:21.718 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:21.719 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:21.719 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:21.719 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:21.719 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:21.719 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:21.719 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:21.719 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:21.719 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:23:21.728 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T06:23:21.728 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:21.728 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:21.728 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:21.728 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:21.728 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:21.729 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:21.729 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:21.729 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:21.729 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:21.729 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:21.729 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:21.729 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:21.729 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:21.729 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:21.729 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:21.729 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:21.729 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:21.729 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:21.729 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:21.730 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:21.730 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:21.730 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:21.730 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:21.730 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:21.730 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:21.730 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:21.730 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:21.730 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:21.730 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:21.730 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:23:21.749 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T06:23:21.758 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T06:23:21.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T06:23:21.886 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:21.939 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:22.061 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:22.168 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:22.224 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:22.350 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:22.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:22.512 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:22.632 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:22.740 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:22.796 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:22.916 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:23.032 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:23.095 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:34.120 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:34.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:34.279 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:34.402 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:34.511 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:34.568 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:34.686 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:34.792 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:34.848 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:34.967 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:35.075 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:35.132 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:35.252 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:35.357 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:35.412 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:35.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:35.690 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:35.761 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:35.905 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:36.030 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:36.085 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:36.204 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:36.311 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:36.364 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:36.486 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:36.595 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:36.656 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:36.778 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:36.884 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:36.940 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:37.060 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:37.168 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:37.222 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:37.338 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:37.443 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:37.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:37.620 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:37.732 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:37.788 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:37.909 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:38.015 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:38.072 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:38.192 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:38.302 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:38.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:38.479 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:38.584 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:38.639 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:38.759 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:38.868 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:38.924 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:39.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:39.157 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:39.211 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:39.333 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:39.439 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:39.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:39.621 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:39.728 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:39.783 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:39.900 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:40.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:40.064 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:40.187 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:40.297 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:40.352 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:40.476 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:40.587 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:40.643 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:40.768 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:53.369 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:53.423 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:53.545 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:53.652 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:53.712 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:53.832 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:23:53.944 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-19 zones={test-zone3, test-zone2} 2021-02-11T06:23:53.944 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:23:53.945 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:23:53.945 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:23:53.995 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:24:06.907 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:24:06.909 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"00000000000000000000:00000000000000004635","last_update":"2021-02-11T06:21:59.639441Z"},{"marker":"00000000000000000000:00000000000000000000","last_update":"0.000000"},{"marker":"00000000000000000000:00000000000000002507","last_update":"2021-02-11T06:21:37.634942Z"},{"marker":"00000000000000000000:00000000000000001044","last_update":"2021-02-11T06:22:43.644624Z"}] 2021-02-11T06:24:06.909 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-02-11T06:24:06.910 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:24:06.910 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:24:06.910 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:24:06.953 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:24:06.953 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:24:07.201 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":15153533235123629405},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004635","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:21:59.639441Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000000","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002507","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:21:37.634942Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000001044","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:22:43.644624Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T06:24:07.202 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-02-11T06:24:07.202 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:24:07.202 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:24:07.202 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:24:07.243 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:24:07.243 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:24:07.497 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"00000000000000000000:00000000000000004635","last_update":"2021-02-11T06:21:59.639441Z"},{"marker":"00000000000000000000:00000000000000000000","last_update":"0.000000"},{"marker":"00000000000000000000:00000000000000002507","last_update":"2021-02-11T06:21:37.634942Z"},{"marker":"00000000000000000000:00000000000000001044","last_update":"2021-02-11T06:22:43.644624Z"}] 2021-02-11T06:24:12.245 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-02-11T06:24:12.245 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:24:12.245 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:24:12.245 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:24:12.291 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:24:12.291 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:24:12.573 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":15040254564068855740},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004635","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:21:59.639441Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000000","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002507","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:21:37.634942Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000001044","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:22:43.644624Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T06:24:12.573 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-02-11T06:24:12.574 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:24:12.574 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:24:12.574 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:24:12.657 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:24:12.657 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:24:12.920 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"00000000000000000000:00000000000000004635","last_update":"2021-02-11T06:21:59.639441Z"},{"marker":"00000000000000000000:00000000000000000000","last_update":"0.000000"},{"marker":"00000000000000000000:00000000000000002507","last_update":"2021-02-11T06:21:37.634942Z"},{"marker":"00000000000000000000:00000000000000001044","last_update":"2021-02-11T06:22:43.644624Z"}] 2021-02-11T06:24:12.921 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-02-11T06:24:12.921 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:24:12.921 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:24:12.921 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:24:12.965 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:24:12.965 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:24:13.221 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":1899655024930128290},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004635","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:21:59.639441Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000000","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002507","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:21:37.634942Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000001044","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:22:43.644624Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T06:24:13.221 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-02-11T06:24:13.221 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:24:13.222 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:24:13.222 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:24:13.311 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:24:13.311 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:24:13.571 INFO:teuthology.orchestra.run.smithi172.stdout:[{"marker":"00000000000000000000:00000000000000004635","last_update":"2021-02-11T06:21:59.791157Z"},{"marker":"00000000000000000000:00000000000000000000","last_update":"0.000000"},{"marker":"00000000000000000000:00000000000000002374","last_update":"2021-02-11T06:21:28.667313Z"},{"marker":"00000000000000000000:00000000000000001044","last_update":"2021-02-11T06:22:43.795890Z"}] 2021-02-11T06:24:13.572 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-02-11T06:24:13.573 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:24:13.573 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:24:13.574 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:24:13.623 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:24:13.623 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:24:13.876 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":3699056310506273018},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004635","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:21:59.791157Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000000","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002374","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:21:28.667313Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000001044","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:22:43.795890Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T06:24:13.877 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-02-11T06:24:13.877 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:24:13.877 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:24:13.877 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:24:13.919 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:24:13.919 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:24:14.157 INFO:teuthology.orchestra.run.smithi172.stdout:[{"marker":"00000000000000000000:00000000000000004635","last_update":"2021-02-11T06:21:59.791157Z"},{"marker":"00000000000000000000:00000000000000000000","last_update":"0.000000"},{"marker":"00000000000000000000:00000000000000002374","last_update":"2021-02-11T06:21:28.667313Z"},{"marker":"00000000000000000000:00000000000000001044","last_update":"2021-02-11T06:22:43.795890Z"}] 2021-02-11T06:24:14.158 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-02-11T06:24:14.158 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:24:14.158 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:24:14.158 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:24:14.195 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:24:14.195 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:24:14.439 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":995053126825359944},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004635","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:21:59.791157Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000000","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002374","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:21:28.667313Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000001044","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:22:43.795890Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T06:24:14.439 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-02-11T06:24:14.439 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:24:14.440 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:24:14.440 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:24:14.520 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:24:14.520 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:24:14.794 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"00000000000000000000:00000000000000004635","last_update":"2021-02-11T06:21:59.817702Z"},{"marker":"00000000000000000000:00000000000000000000","last_update":"0.000000"},{"marker":"00000000000000000000:00000000000000002374","last_update":"2021-02-11T06:21:28.930530Z"},{"marker":"00000000000000000000:00000000000000001044","last_update":"2021-02-11T06:22:43.821844Z"}] 2021-02-11T06:24:14.794 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-02-11T06:24:14.794 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:24:14.795 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:24:14.795 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:24:14.877 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:24:14.878 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:24:15.127 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":398085786125260623},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004635","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:21:59.817702Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000000","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002374","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:21:28.930530Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000001044","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:22:43.821844Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T06:24:15.128 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-02-11T06:24:15.128 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:24:15.128 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:24:15.128 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:24:15.209 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:24:15.209 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:24:15.473 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"00000000000000000000:00000000000000004635","last_update":"2021-02-11T06:21:59.817702Z"},{"marker":"00000000000000000000:00000000000000000000","last_update":"0.000000"},{"marker":"00000000000000000000:00000000000000002374","last_update":"2021-02-11T06:21:28.930530Z"},{"marker":"00000000000000000000:00000000000000001044","last_update":"2021-02-11T06:22:43.821844Z"}] 2021-02-11T06:24:15.474 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-02-11T06:24:15.474 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:24:15.474 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:24:15.474 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:24:15.517 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:24:15.517 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:24:15.767 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":11837782300501369896},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004635","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:21:59.817702Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000000","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002374","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:21:28.930530Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000001044","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:22:43.821844Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T06:24:15.768 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-02-11T06:24:15.768 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_concurrent_versioned_object_incremental_sync ... ok 2021-02-11T06:24:15.809 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:24:15.810 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:24:15.810 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:24:15.847 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:24:15.847 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:24:16.070 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=7c4a812f-81e4-4a72-acce-0ee18589648c 2021-02-11T06:24:16.105 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613024655.802479_156.1","last_update":"2021-02-11T06:24:15.802479Z"},{"marker":"1_1613024301.142235_74.1","last_update":"2021-02-11T06:18:21.142235Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:24:16.105 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T06:24:16.105 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:24:16.105 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:24:16.106 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:24:16.146 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:24:16.146 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:24:16.385 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7c4a812f-81e4-4a72-acce-0ee18589648c","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613024655.802479_156.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:24:15.802479Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:24:16.386 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T06:24:16.386 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T06:24:16.386 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:24:16.386 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:24:16.387 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:24:16.428 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:24:16.428 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:24:16.687 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7c4a812f-81e4-4a72-acce-0ee18589648c","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613024655.802479_156.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:24:15.802479Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:24:16.687 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T06:24:16.687 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:24:16.688 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:24:16.688 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:24:16.688 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:24:16.727 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:24:16.727 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:24:16.990 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7c4a812f-81e4-4a72-acce-0ee18589648c","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613024655.802479_156.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:24:15.802479Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:24:16.991 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:24:17.043 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:24:17.043 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:24:17.043 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:24:17.092 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:24:17.092 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:24:17.317 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:24:17.312+0000 7fe02b86b080 1 bucket sync caught up with source: 2021-02-11T06:24:17.317 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , , , , 00000000001.23.5, , ] 2021-02-11T06:24:17.317 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , , , , 00000000001.23.5, , ] 2021-02-11T06:24:17.318 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:24:17.312+0000 7fe02b86b080 0 bucket checkpoint complete 2021-02-11T06:24:17.349 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:24:17.349 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:24:17.350 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:24:17.400 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:24:17.401 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:24:17.644 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:24:17.641+0000 7f2c49d93080 1 bucket sync caught up with source: 2021-02-11T06:24:17.644 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, , , , , , , , 00000000001.23.5, , ] 2021-02-11T06:24:17.644 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, , , , , , , , 00000000001.23.5, , ] 2021-02-11T06:24:17.644 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:24:17.641+0000 7f2c49d93080 0 bucket checkpoint complete 2021-02-11T06:24:17.672 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:24:17.672 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:24:17.673 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:24:17.718 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:24:17.718 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:24:17.953 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:24:17.948+0000 7f8e17a53080 1 bucket sync caught up with source: 2021-02-11T06:24:17.954 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , , , , 00000000001.23.5, , ] 2021-02-11T06:24:17.954 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , , , , 00000000001.23.5, , ] 2021-02-11T06:24:17.954 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:24:17.948+0000 7f8e17a53080 0 bucket checkpoint complete 2021-02-11T06:24:17.984 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:24:17.985 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:24:17.985 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:24:18.033 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:24:18.033 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:24:18.271 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:24:18.269+0000 7ff6bc5f8080 1 bucket sync caught up with source: 2021-02-11T06:24:18.271 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, , , , , , , , 00000000001.23.5, , ] 2021-02-11T06:24:18.271 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, , , , , , , , 00000000001.23.5, , ] 2021-02-11T06:24:18.271 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:24:18.269+0000 7ff6bc5f8080 0 bucket checkpoint complete 2021-02-11T06:24:18.305 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:24:18.305 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:24:18.305 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:24:18.351 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:24:18.351 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:24:18.597 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:24:18.597+0000 7f9ff1fed080 1 bucket sync caught up with source: 2021-02-11T06:24:18.597 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, , , , , , , , 00000000001.23.5, , ] 2021-02-11T06:24:18.598 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, , , , , , , , 00000000001.23.5, , ] 2021-02-11T06:24:18.598 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:24:18.597+0000 7f9ff1fed080 0 bucket checkpoint complete 2021-02-11T06:24:18.629 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:24:18.629 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:24:18.629 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:24:18.676 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:24:18.676 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:24:18.922 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:24:18.921+0000 7f808cbd7080 1 bucket sync caught up with source: 2021-02-11T06:24:18.922 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, , , , , , , , 00000000001.37.5, , ] 2021-02-11T06:24:18.922 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, , , , , , , , 00000000001.37.5, , ] 2021-02-11T06:24:18.922 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:24:18.921+0000 7f808cbd7080 0 bucket checkpoint complete 2021-02-11T06:24:18.953 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:24:18.953 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:24:18.954 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:24:19.002 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:24:19.002 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:24:19.224 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:24:19.216+0000 7fcbfdf8a080 1 bucket sync caught up with source: 2021-02-11T06:24:19.224 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , , , , 00000000001.37.5, , ] 2021-02-11T06:24:19.224 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , , , , 00000000001.37.5, , ] 2021-02-11T06:24:19.224 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:24:19.216+0000 7fcbfdf8a080 0 bucket checkpoint complete 2021-02-11T06:24:19.250 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-20 zones={test-zone2, test-zone1} 2021-02-11T06:24:19.261 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:24:19.307 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:24:19.307 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:24:19.314 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:24:19.368 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:24:19.379 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:24:19.448 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:24:19.510 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-20 zones={test-zone2, test-zone1} 2021-02-11T06:24:19.510 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-20 zones={test-zone3, test-zone1} 2021-02-11T06:24:19.519 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:24:19.566 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:24:19.567 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:24:19.573 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:24:19.628 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:24:19.639 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:24:19.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:24:19.760 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-20 zones={test-zone3, test-zone1} 2021-02-11T06:24:19.761 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-20 zones={test-zone3, test-zone2} 2021-02-11T06:24:19.770 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:24:19.819 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:24:19.819 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:24:19.826 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:24:19.882 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:24:19.893 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:24:19.959 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:24:20.017 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-20 zones={test-zone3, test-zone2} 2021-02-11T06:24:20.033 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:24:20.033 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:24:20.033 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:24:20.069 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:24:20.069 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:24:20.268 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=7c4a812f-81e4-4a72-acce-0ee18589648c 2021-02-11T06:24:46.611 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613024660.030516_162.1","last_update":"2021-02-11T06:24:20.030516Z"},{"marker":"1_1613024301.142235_74.1","last_update":"2021-02-11T06:18:21.142235Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:24:46.612 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T06:24:46.612 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:24:46.612 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:24:46.612 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:24:46.653 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:24:46.653 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:24:46.899 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7c4a812f-81e4-4a72-acce-0ee18589648c","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613024660.030516_162.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:24:20.030516Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:24:46.900 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T06:24:46.900 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T06:24:46.900 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:24:46.900 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:24:46.900 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:24:46.942 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:24:46.942 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:24:47.207 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7c4a812f-81e4-4a72-acce-0ee18589648c","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613024660.030516_162.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:24:20.030516Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:24:47.207 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T06:24:47.207 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:24:47.207 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:24:47.208 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:24:47.208 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:24:47.247 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:24:47.247 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:24:47.506 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7c4a812f-81e4-4a72-acce-0ee18589648c","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613024660.030516_162.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:24:20.030516Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:24:47.507 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:24:47.559 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:24:47.559 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:24:47.559 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:24:47.605 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:24:47.606 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:24:47.852 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:24:47.843+0000 7fe7468ee080 1 bucket sync caught up with source: 2021-02-11T06:24:47.852 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , , , , 00000000003.26.12, , ] 2021-02-11T06:24:47.853 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , , , , 00000000003.26.12, , ] 2021-02-11T06:24:47.853 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:24:47.843+0000 7fe7468ee080 0 bucket checkpoint complete 2021-02-11T06:24:47.877 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:24:47.878 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:24:47.878 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:24:47.925 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:24:47.925 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:24:48.152 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:24:48.148+0000 7f74b0718080 1 bucket sync caught up with source: 2021-02-11T06:24:48.153 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, , , , , , , , 00000000003.26.12, , ] 2021-02-11T06:24:48.153 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, , , , , , , , 00000000003.26.12, , ] 2021-02-11T06:24:48.153 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:24:48.148+0000 7f74b0718080 0 bucket checkpoint complete 2021-02-11T06:24:48.180 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:24:48.180 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:24:48.181 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:24:48.229 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:24:48.229 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:24:48.450 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:24:48.443+0000 7fa85a6a3080 1 bucket sync caught up with source: 2021-02-11T06:24:48.450 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , , , , 00000000003.26.12, , ] 2021-02-11T06:24:48.450 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , , , , 00000000003.26.12, , ] 2021-02-11T06:24:48.450 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:24:48.443+0000 7fa85a6a3080 0 bucket checkpoint complete 2021-02-11T06:24:48.478 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:24:48.478 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:24:48.478 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:24:48.525 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:24:48.526 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:24:48.760 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:24:48.756+0000 7f0de2c67080 1 bucket sync caught up with source: 2021-02-11T06:24:48.761 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, , , , , , , , 00000000004.30.9, , ] 2021-02-11T06:24:48.761 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, , , , , , , , 00000000004.30.9, , ] 2021-02-11T06:24:48.761 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:24:48.756+0000 7f0de2c67080 0 bucket checkpoint complete 2021-02-11T06:24:48.793 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:24:48.793 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:24:48.794 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:24:48.840 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:24:48.840 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:24:49.092 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:24:49.088+0000 7f66bfbf6080 1 bucket sync caught up with source: 2021-02-11T06:24:49.092 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, , , , , , , , 00000000004.30.9, , ] 2021-02-11T06:24:49.092 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, , , , , , , , 00000000004.30.9, , ] 2021-02-11T06:24:49.092 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:24:49.088+0000 7f66bfbf6080 0 bucket checkpoint complete 2021-02-11T06:24:49.126 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:24:49.126 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:24:49.126 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:24:49.172 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:24:49.172 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:24:49.401 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:24:49.400+0000 7fb13d9a8080 1 bucket sync caught up with source: 2021-02-11T06:24:49.401 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, , , , , , , , 00000000004.44.9, , ] 2021-02-11T06:24:49.401 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, , , , , , , , 00000000004.44.9, , ] 2021-02-11T06:24:49.401 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:24:49.400+0000 7fb13d9a8080 0 bucket checkpoint complete 2021-02-11T06:24:49.434 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:24:49.434 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:24:49.434 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:24:49.481 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:24:49.482 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:24:49.722 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:24:49.715+0000 7fa88347c080 1 bucket sync caught up with source: 2021-02-11T06:24:49.723 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , , , , 00000000004.44.9, , ] 2021-02-11T06:24:49.723 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , , , , 00000000004.44.9, , ] 2021-02-11T06:24:49.723 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:24:49.715+0000 7fa88347c080 0 bucket checkpoint complete 2021-02-11T06:24:49.756 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-20 zones={test-zone2, test-zone1} 2021-02-11T06:24:49.766 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:24:49.819 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:24:49.819 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:24:49.819 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:24:49.826 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:24:49.826 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:24:49.882 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:24:49.893 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:24:49.961 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:24:50.022 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:24:50.034 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:24:50.102 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:24:50.163 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-20 zones={test-zone2, test-zone1} 2021-02-11T06:24:50.163 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-20 zones={test-zone3, test-zone1} 2021-02-11T06:24:50.173 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:24:50.225 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:24:51.299 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:24:51.299 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:24:51.307 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:24:51.307 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:24:51.358 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:24:51.369 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:24:51.436 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:24:51.497 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:24:51.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:24:51.572 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:24:51.629 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-20 zones={test-zone3, test-zone1} 2021-02-11T06:24:51.629 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-20 zones={test-zone3, test-zone2} 2021-02-11T06:24:51.639 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:24:51.686 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:24:51.687 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:24:51.687 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:24:51.694 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:24:51.694 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:24:51.750 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:24:51.761 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:24:51.829 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:24:51.891 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:24:51.903 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:24:51.969 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:24:52.029 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-20 zones={test-zone3, test-zone2} 2021-02-11T06:24:52.047 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:24:52.047 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:24:52.047 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:24:52.087 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:24:52.088 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:24:52.290 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=7c4a812f-81e4-4a72-acce-0ee18589648c 2021-02-11T06:24:52.325 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613024692.042180_169.1","last_update":"2021-02-11T06:24:52.042180Z"},{"marker":"1_1613024301.142235_74.1","last_update":"2021-02-11T06:18:21.142235Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:24:52.325 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T06:24:52.325 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:24:52.325 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:24:52.326 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:24:52.365 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:24:52.365 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:24:52.623 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7c4a812f-81e4-4a72-acce-0ee18589648c","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613024692.042180_169.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:24:52.042180Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:24:52.623 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T06:24:52.623 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T06:24:52.623 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:24:52.623 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:24:52.624 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:24:52.663 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:24:52.663 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:24:52.928 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7c4a812f-81e4-4a72-acce-0ee18589648c","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613024692.042180_169.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:24:52.042180Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:25:08.924 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T06:25:08.925 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:25:08.925 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:25:08.925 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:25:08.925 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:25:08.970 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:25:08.970 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:25:09.230 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7c4a812f-81e4-4a72-acce-0ee18589648c","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613024692.042180_169.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:24:52.042180Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:25:09.231 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:25:09.283 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:09.283 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:09.284 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:25:09.328 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:25:09.329 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:25:09.558 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:25:09.555+0000 7fa7c79e4080 1 bucket sync caught up with source: 2021-02-11T06:25:09.558 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , , , , 00000000005.30.17, , ] 2021-02-11T06:25:09.558 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , , , , 00000000005.30.17, , ] 2021-02-11T06:25:09.558 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:25:09.555+0000 7fa7c79e4080 0 bucket checkpoint complete 2021-02-11T06:25:09.588 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:09.588 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:09.588 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:25:09.639 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:25:09.639 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:25:09.858 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:25:09.856+0000 7fc49d3cb080 1 bucket sync caught up with source: 2021-02-11T06:25:09.858 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, , , , , , , , 00000000005.30.17, , ] 2021-02-11T06:25:09.859 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, , , , , , , , 00000000005.30.17, , ] 2021-02-11T06:25:09.859 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:25:09.856+0000 7fc49d3cb080 0 bucket checkpoint complete 2021-02-11T06:25:09.885 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:09.885 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:09.885 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:25:09.933 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:25:09.933 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:25:10.169 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:25:10.167+0000 7fe74470c080 1 bucket sync caught up with source: 2021-02-11T06:25:10.170 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , , , , 00000000005.30.17, , ] 2021-02-11T06:25:10.170 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , , , , 00000000005.30.17, , ] 2021-02-11T06:25:10.170 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:25:10.167+0000 7fe74470c080 0 bucket checkpoint complete 2021-02-11T06:25:10.199 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:10.199 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:10.200 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:25:10.249 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:25:10.249 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:25:10.508 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:25:10.504+0000 7f521c3fe080 1 bucket sync caught up with source: 2021-02-11T06:25:10.508 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, , , , , , , , 00000000007.38.14, , ] 2021-02-11T06:25:10.508 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, , , , , , , , 00000000007.38.14, , ] 2021-02-11T06:25:10.508 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:25:10.504+0000 7f521c3fe080 0 bucket checkpoint complete 2021-02-11T06:25:10.540 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:10.540 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:10.541 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:25:10.587 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:25:10.588 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:25:10.832 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:25:10.828+0000 7f768deec080 1 bucket sync caught up with source: 2021-02-11T06:25:10.832 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, , , , , , , , 00000000007.38.14, , ] 2021-02-11T06:25:10.832 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, , , , , , , , 00000000007.38.14, , ] 2021-02-11T06:25:10.832 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:25:10.828+0000 7f768deec080 0 bucket checkpoint complete 2021-02-11T06:25:10.863 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:10.863 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:10.863 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:25:10.907 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:25:10.907 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:25:11.150 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:25:11.148+0000 7f2084162080 1 bucket sync caught up with source: 2021-02-11T06:25:11.150 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, , , , , , , , 00000000007.52.14, , ] 2021-02-11T06:25:11.150 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, , , , , , , , 00000000007.52.14, , ] 2021-02-11T06:25:11.151 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:25:11.148+0000 7f2084162080 0 bucket checkpoint complete 2021-02-11T06:25:11.176 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:11.176 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:11.176 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:25:11.222 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:25:11.222 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:25:11.437 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:25:11.431+0000 7fb835912080 1 bucket sync caught up with source: 2021-02-11T06:25:11.437 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , , , , 00000000007.52.14, , ] 2021-02-11T06:25:11.437 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , , , , 00000000007.52.14, , ] 2021-02-11T06:25:11.437 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:25:11.431+0000 7fb835912080 0 bucket checkpoint complete 2021-02-11T06:25:11.469 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-20 zones={test-zone2, test-zone1} 2021-02-11T06:25:11.480 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:25:11.531 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:25:11.531 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:25:11.531 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:25:11.538 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:25:11.538 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:25:11.594 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:25:11.606 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:25:11.671 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:25:11.729 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:25:11.740 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:25:11.808 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:25:11.870 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-20 zones={test-zone2, test-zone1} 2021-02-11T06:25:11.870 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-20 zones={test-zone3, test-zone1} 2021-02-11T06:25:11.879 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:25:11.926 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:25:11.927 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:25:11.927 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:25:11.932 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:25:11.932 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:25:11.986 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:25:11.997 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:25:12.063 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:25:12.124 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:25:12.133 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:25:12.200 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:25:12.265 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-20 zones={test-zone3, test-zone1} 2021-02-11T06:25:12.265 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-20 zones={test-zone3, test-zone2} 2021-02-11T06:25:12.277 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:25:12.327 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:25:12.327 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:25:12.327 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:25:12.334 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:25:12.335 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:25:12.389 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:25:12.401 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:25:12.466 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:25:12.525 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:25:12.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:25:12.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:25:12.666 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-20 zones={test-zone3, test-zone2} 2021-02-11T06:25:12.667 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_version_suspended_incremental_sync ... ok 2021-02-11T06:25:12.667 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xscaju-21 2021-02-11T06:25:12.706 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xscaju-22 2021-02-11T06:25:12.769 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xscaju-23 2021-02-11T06:25:12.898 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:25:12.898 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:25:12.898 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:25:12.934 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:25:12.934 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:25:13.153 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=7c4a812f-81e4-4a72-acce-0ee18589648c 2021-02-11T06:25:13.184 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613024712.893011_191.1","last_update":"2021-02-11T06:25:12.893011Z"},{"marker":"1_1613024301.142235_74.1","last_update":"2021-02-11T06:18:21.142235Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:25:13.184 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T06:25:13.184 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:25:13.184 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:25:13.185 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:25:13.226 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:25:13.226 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:25:13.459 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7c4a812f-81e4-4a72-acce-0ee18589648c","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613024712.893011_191.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:25:12.893011Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:25:13.459 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T06:25:13.459 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T06:25:13.459 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:25:13.460 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:25:13.460 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:25:13.501 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:25:13.501 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:25:13.760 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7c4a812f-81e4-4a72-acce-0ee18589648c","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613024712.893011_191.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:25:12.893011Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:25:13.760 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T06:25:13.760 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:25:13.760 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:25:13.760 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:25:13.761 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:25:13.800 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:25:13.800 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:25:14.067 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7c4a812f-81e4-4a72-acce-0ee18589648c","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613024712.893011_191.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:25:12.893011Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:25:14.068 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:25:14.294 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:14.294 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:14.294 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:25:14.339 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:25:14.340 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:25:14.559 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:25:14.554+0000 7f857a919080 1 bucket sync caught up with source: 2021-02-11T06:25:14.559 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , , , , 00000000003.19.11, , ] 2021-02-11T06:25:14.559 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , , , , 00000000003.19.11, , ] 2021-02-11T06:25:14.559 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:25:14.554+0000 7f857a919080 0 bucket checkpoint complete 2021-02-11T06:25:14.591 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:14.591 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:14.591 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:25:14.640 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:25:14.640 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:25:14.860 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:25:14.856+0000 7f5c77cc0080 1 bucket sync caught up with source: 2021-02-11T06:25:14.860 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, , , , , , , , 00000000003.19.11, , ] 2021-02-11T06:25:14.860 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, , , , , , , , 00000000003.19.11, , ] 2021-02-11T06:25:14.860 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:25:14.856+0000 7f5c77cc0080 0 bucket checkpoint complete 2021-02-11T06:25:14.886 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:14.887 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:14.887 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:25:14.935 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:25:14.935 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:25:15.156 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:25:15.150+0000 7f8ca45f4080 1 bucket sync caught up with source: 2021-02-11T06:25:15.156 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , , , , 00000000003.19.11, , ] 2021-02-11T06:25:15.157 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , , , , 00000000003.19.11, , ] 2021-02-11T06:25:15.157 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:25:15.150+0000 7f8ca45f4080 0 bucket checkpoint complete 2021-02-11T06:25:15.187 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:15.187 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:15.187 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-21 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:25:15.234 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:25:15.234 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:25:15.469 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:25:15.468+0000 7faa03392080 1 bucket sync caught up with source: 2021-02-11T06:25:15.469 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, , , , , , , , 00000000002.36.5, , ] 2021-02-11T06:25:15.470 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, , , , , , , , 00000000002.36.5, , ] 2021-02-11T06:25:15.470 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:25:15.468+0000 7faa03392080 0 bucket checkpoint complete 2021-02-11T06:25:15.509 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:15.509 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:15.509 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-21 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:25:15.554 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:25:15.554 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:25:15.804 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:25:15.800+0000 7f0c2c1b0080 1 bucket sync caught up with source: 2021-02-11T06:25:15.804 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, , , , , , , , 00000000002.36.5, , ] 2021-02-11T06:25:15.804 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, , , , , , , , 00000000002.36.5, , ] 2021-02-11T06:25:15.804 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:25:15.800+0000 7f0c2c1b0080 0 bucket checkpoint complete 2021-02-11T06:25:15.835 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:15.835 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:15.835 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:25:15.883 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:25:15.883 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:25:16.121 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:25:16.120+0000 7f3f47ea5080 1 bucket sync caught up with source: 2021-02-11T06:25:16.122 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, , , , , , , , 00000000002.28.5, , ] 2021-02-11T06:25:16.122 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, , , , , , , , 00000000002.28.5, , ] 2021-02-11T06:25:16.122 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:25:16.120+0000 7f3f47ea5080 0 bucket checkpoint complete 2021-02-11T06:25:16.155 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:16.156 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:16.156 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:25:16.202 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:25:16.202 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:25:16.451 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:25:16.446+0000 7f5476468080 1 bucket sync caught up with source: 2021-02-11T06:25:16.451 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , , , , 00000000002.28.5, , ] 2021-02-11T06:25:16.451 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , , , , 00000000002.28.5, , ] 2021-02-11T06:25:16.451 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:25:16.446+0000 7f5476468080 0 bucket checkpoint complete 2021-02-11T06:25:16.482 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-21 zones={test-zone2, test-zone1} 2021-02-11T06:25:16.492 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:25:16.539 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:25:16.539 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:25:16.539 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:25:16.546 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:25:16.546 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:25:16.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:25:16.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:25:16.622 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:25:16.690 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-21 zones={test-zone2, test-zone1} 2021-02-11T06:25:16.691 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-21 zones={test-zone3, test-zone1} 2021-02-11T06:25:16.701 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:25:16.747 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:25:16.747 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:25:16.747 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:25:16.753 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:25:16.753 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:25:16.810 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:25:16.818 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:25:16.828 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:25:16.893 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-21 zones={test-zone3, test-zone1} 2021-02-11T06:25:16.894 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-21 zones={test-zone3, test-zone2} 2021-02-11T06:25:16.904 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:25:16.951 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:25:16.951 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:25:16.951 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:25:16.958 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:25:16.958 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:25:17.013 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:25:17.021 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:25:17.032 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:25:17.101 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-21 zones={test-zone3, test-zone2} 2021-02-11T06:25:17.102 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:17.102 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:17.102 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:25:17.145 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:25:17.145 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:25:17.391 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:25:17.386+0000 7fba36575080 1 bucket sync caught up with source: 2021-02-11T06:25:17.391 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , , , , 00000000002.7.5, , ] 2021-02-11T06:25:17.391 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , , , , 00000000002.7.5, , ] 2021-02-11T06:25:17.391 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:25:17.386+0000 7fba36575080 0 bucket checkpoint complete 2021-02-11T06:25:17.422 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:17.422 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:17.422 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:25:17.471 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:25:17.471 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:25:17.702 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:25:17.700+0000 7f722f8c4080 1 bucket sync caught up with source: 2021-02-11T06:25:17.702 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, , , , , , , , 00000000002.7.5, , ] 2021-02-11T06:25:17.702 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, , , , , , , , 00000000002.7.5, , ] 2021-02-11T06:25:17.702 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:25:17.700+0000 7f722f8c4080 0 bucket checkpoint complete 2021-02-11T06:25:17.730 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:17.730 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:17.731 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:25:17.777 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:25:17.777 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:25:18.002 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:25:17.998+0000 7efdf0f81080 1 bucket sync caught up with source: 2021-02-11T06:25:25.685 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , , , , 00000000002.7.5, , ] 2021-02-11T06:25:25.685 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , , , , 00000000002.7.5, , ] 2021-02-11T06:25:25.686 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:25:17.998+0000 7efdf0f81080 0 bucket checkpoint complete 2021-02-11T06:25:25.687 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:25.687 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:25.688 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-22 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:25:25.734 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:25:25.734 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:25:25.995 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:25:25.992+0000 7fce088eb080 1 bucket sync caught up with source: 2021-02-11T06:25:25.995 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, , , , , , , , 00000000003.12.11, , ] 2021-02-11T06:25:25.995 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, , , , , , , , 00000000003.12.11, , ] 2021-02-11T06:25:25.995 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:25:25.992+0000 7fce088eb080 0 bucket checkpoint complete 2021-02-11T06:25:26.029 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:26.029 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:26.029 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-22 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:25:26.076 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:25:26.076 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:25:26.312 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:25:26.308+0000 7fc2fecd8080 1 bucket sync caught up with source: 2021-02-11T06:25:26.312 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, , , , , , , , 00000000003.12.11, , ] 2021-02-11T06:25:26.312 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, , , , , , , , 00000000003.12.11, , ] 2021-02-11T06:25:26.313 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:25:26.308+0000 7fc2fecd8080 0 bucket checkpoint complete 2021-02-11T06:25:26.344 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:26.344 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:26.344 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-22 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:25:26.390 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:25:26.390 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:25:26.629 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:25:26.628+0000 7f3ce0ca1080 1 bucket sync caught up with source: 2021-02-11T06:25:26.630 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, , , , , , , , 00000000002.12.5, , ] 2021-02-11T06:25:26.630 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, , , , , , , , 00000000002.12.5, , ] 2021-02-11T06:25:26.630 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:25:26.628+0000 7f3ce0ca1080 0 bucket checkpoint complete 2021-02-11T06:25:26.659 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:26.660 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:26.660 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-22 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:25:26.707 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:25:26.707 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:25:26.937 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:25:26.930+0000 7fc2161d2080 1 bucket sync caught up with source: 2021-02-11T06:25:26.937 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , , , , 00000000002.12.5, , ] 2021-02-11T06:25:26.937 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , , , , 00000000002.12.5, , ] 2021-02-11T06:25:26.937 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:25:26.930+0000 7fc2161d2080 0 bucket checkpoint complete 2021-02-11T06:25:26.967 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-22 zones={test-zone2, test-zone1} 2021-02-11T06:25:26.978 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:25:27.027 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:25:27.027 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:25:27.027 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:25:27.035 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:25:27.035 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:25:27.090 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:25:27.098 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:25:27.111 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:25:27.180 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-22 zones={test-zone2, test-zone1} 2021-02-11T06:25:27.180 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-22 zones={test-zone3, test-zone1} 2021-02-11T06:25:27.190 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:25:27.239 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:25:27.239 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:25:27.239 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:25:27.246 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:25:27.247 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:25:27.301 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:25:27.309 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:25:27.319 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:25:27.386 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-22 zones={test-zone3, test-zone1} 2021-02-11T06:25:27.387 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-22 zones={test-zone3, test-zone2} 2021-02-11T06:25:27.395 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:25:27.443 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:25:27.443 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:25:27.443 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:25:27.450 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:25:27.450 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:25:27.506 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:25:27.514 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:25:27.527 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:25:27.598 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-22 zones={test-zone3, test-zone2} 2021-02-11T06:25:27.598 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:27.598 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:27.599 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:25:27.644 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:25:27.644 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:25:27.893 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:25:27.886+0000 7fd292836080 1 bucket sync caught up with source: 2021-02-11T06:25:27.893 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , , , , 00000000002.11.5, , ] 2021-02-11T06:25:27.893 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , , , , 00000000002.11.5, , ] 2021-02-11T06:25:27.893 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:25:27.886+0000 7fd292836080 0 bucket checkpoint complete 2021-02-11T06:25:27.925 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:27.925 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:27.925 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:25:27.974 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:25:27.974 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:25:28.220 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:25:28.215+0000 7f64c407e080 1 bucket sync caught up with source: 2021-02-11T06:25:28.221 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, , , , , , , , 00000000002.11.5, , ] 2021-02-11T06:25:28.221 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, , , , , , , , 00000000002.11.5, , ] 2021-02-11T06:25:28.221 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:25:28.219+0000 7f64c407e080 0 bucket checkpoint complete 2021-02-11T06:25:28.247 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:28.247 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:28.247 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:25:28.295 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:25:28.295 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:25:28.544 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:25:28.538+0000 7f604bdcc080 1 bucket sync caught up with source: 2021-02-11T06:25:28.544 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , , , , 00000000002.11.5, , ] 2021-02-11T06:25:28.544 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , , , , 00000000002.11.5, , ] 2021-02-11T06:25:28.544 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:25:28.538+0000 7f604bdcc080 0 bucket checkpoint complete 2021-02-11T06:25:28.576 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:28.576 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:28.576 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:25:28.625 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:25:28.625 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:25:28.876 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:25:28.871+0000 7fef3ab4e080 1 bucket sync caught up with source: 2021-02-11T06:25:28.876 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, , , , , , , , 00000000002.17.5, , ] 2021-02-11T06:25:28.876 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, , , , , , , , 00000000002.17.5, , ] 2021-02-11T06:25:28.876 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:25:28.871+0000 7fef3ab4e080 0 bucket checkpoint complete 2021-02-11T06:25:28.910 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:28.910 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:28.910 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:25:28.956 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:25:28.956 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:25:29.206 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:25:29.203+0000 7f80df914080 1 bucket sync caught up with source: 2021-02-11T06:25:29.206 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, , , , , , , , 00000000002.17.5, , ] 2021-02-11T06:25:29.206 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, , , , , , , , 00000000002.17.5, , ] 2021-02-11T06:25:29.207 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:25:29.203+0000 7f80df914080 0 bucket checkpoint complete 2021-02-11T06:25:29.239 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:29.239 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:29.239 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:25:29.288 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:25:29.288 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:25:29.549 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:25:29.547+0000 7fdeb770f080 1 bucket sync caught up with source: 2021-02-11T06:25:29.549 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, , , , , , , , 00000000003.20.11, , ] 2021-02-11T06:25:29.549 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, , , , , , , , 00000000003.20.11, , ] 2021-02-11T06:25:29.549 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:25:29.547+0000 7fdeb770f080 0 bucket checkpoint complete 2021-02-11T06:25:29.576 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:29.576 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:29.577 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:25:29.622 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:25:29.623 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:25:29.873 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:25:29.866+0000 7f473e06b080 1 bucket sync caught up with source: 2021-02-11T06:25:29.873 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , , , , 00000000003.20.11, , ] 2021-02-11T06:25:29.873 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , , , , 00000000003.20.11, , ] 2021-02-11T06:25:29.873 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:25:29.866+0000 7f473e06b080 0 bucket checkpoint complete 2021-02-11T06:25:29.903 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-23 zones={test-zone2, test-zone1} 2021-02-11T06:25:29.913 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:25:29.963 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:25:29.963 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:25:29.963 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:25:29.971 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:25:29.971 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:25:30.025 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:25:30.032 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:25:30.043 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:25:30.110 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-23 zones={test-zone2, test-zone1} 2021-02-11T06:25:30.110 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-23 zones={test-zone3, test-zone1} 2021-02-11T06:25:30.120 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:25:30.167 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:25:30.167 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:25:30.167 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:25:30.174 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:25:30.174 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:25:30.230 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:25:30.238 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:25:30.247 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:25:30.313 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-23 zones={test-zone3, test-zone1} 2021-02-11T06:25:30.582 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-23 zones={test-zone3, test-zone2} 2021-02-11T06:25:30.593 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:25:30.643 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:25:30.643 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:25:30.643 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:25:30.651 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:25:30.651 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:25:30.705 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:25:30.714 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:25:30.725 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:25:30.792 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-23 zones={test-zone3, test-zone2} 2021-02-11T06:25:30.793 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_delete_marker_full_sync ... ok 2021-02-11T06:25:30.793 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xscaju-24 2021-02-11T06:25:30.832 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xscaju-25 2021-02-11T06:25:30.894 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xscaju-26 2021-02-11T06:25:31.086 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:25:31.086 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:25:31.087 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:25:31.117 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:25:31.118 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:25:31.330 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=7c4a812f-81e4-4a72-acce-0ee18589648c 2021-02-11T06:25:31.370 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613024731.075841_236.1","last_update":"2021-02-11T06:25:31.075841Z"},{"marker":"1_1613024301.142235_74.1","last_update":"2021-02-11T06:18:21.142235Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:25:31.370 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T06:25:31.370 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:25:31.370 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:25:31.371 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:25:31.411 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:25:31.411 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:25:31.661 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7c4a812f-81e4-4a72-acce-0ee18589648c","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613024731.075841_236.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:25:31.075841Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:25:31.662 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T06:25:31.662 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T06:25:31.662 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:25:31.662 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:25:31.662 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:25:31.703 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:25:31.703 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:25:31.984 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7c4a812f-81e4-4a72-acce-0ee18589648c","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613024731.075841_236.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:25:31.075841Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:25:31.985 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T06:25:31.985 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:25:31.985 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:25:31.985 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:25:31.985 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:25:32.024 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:25:32.024 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:25:32.272 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7c4a812f-81e4-4a72-acce-0ee18589648c","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613024731.075841_236.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:25:31.075841Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:25:32.272 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:25:32.510 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:32.510 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:32.510 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:25:32.559 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:25:32.559 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:25:32.755 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:25:32.750+0000 7f526563f080 1 bucket sync caught up with source: 2021-02-11T06:25:32.756 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , , , , 00000000003.11.6, , ] 2021-02-11T06:25:32.756 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , , , , 00000000003.11.6, , ] 2021-02-11T06:25:32.756 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:25:32.750+0000 7f526563f080 0 bucket checkpoint complete 2021-02-11T06:25:32.784 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:32.784 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:32.784 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:25:32.833 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:25:32.833 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:25:33.059 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:25:33.055+0000 7fb31c7db080 1 bucket sync caught up with source: 2021-02-11T06:25:33.059 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, , , , , , , , 00000000003.11.6, , ] 2021-02-11T06:25:33.059 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, , , , , , , , 00000000003.11.6, , ] 2021-02-11T06:25:33.059 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:25:33.055+0000 7fb31c7db080 0 bucket checkpoint complete 2021-02-11T06:25:33.087 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:33.087 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:33.087 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:25:33.134 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:25:33.134 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:25:33.378 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:25:33.374+0000 7f85d5d79080 1 bucket sync caught up with source: 2021-02-11T06:25:33.379 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , , , , 00000000003.11.6, , ] 2021-02-11T06:25:33.379 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , , , , 00000000003.11.6, , ] 2021-02-11T06:25:33.379 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:25:33.374+0000 7f85d5d79080 0 bucket checkpoint complete 2021-02-11T06:25:33.409 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:33.409 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:33.409 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-24 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:25:33.457 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:25:33.458 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:25:33.707 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:25:33.703+0000 7fa0e48b5080 1 bucket sync caught up with source: 2021-02-11T06:25:33.707 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, , , , , , , , 00000000001.9.12, , ] 2021-02-11T06:25:33.708 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, , , , , , , , 00000000001.9.12, , ] 2021-02-11T06:25:33.708 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:25:33.703+0000 7fa0e48b5080 0 bucket checkpoint complete 2021-02-11T06:25:33.742 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:33.742 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:33.742 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-24 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:25:33.800 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:25:33.800 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:25:34.033 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:25:34.031+0000 7f5ed51fe080 1 bucket sync caught up with source: 2021-02-11T06:25:34.033 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, , , , , , , , 00000000001.9.12, , ] 2021-02-11T06:25:34.034 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, , , , , , , , 00000000001.9.12, , ] 2021-02-11T06:25:34.034 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:25:34.031+0000 7f5ed51fe080 0 bucket checkpoint complete 2021-02-11T06:25:34.067 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:34.067 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:34.067 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-24 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:25:34.116 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:25:34.116 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:25:34.351 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:25:34.347+0000 7f627d422080 1 bucket sync caught up with source: 2021-02-11T06:25:34.352 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, , , , , , , , 00000000001.25.12, , ] 2021-02-11T06:25:34.352 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, , , , , , , , 00000000001.25.12, , ] 2021-02-11T06:25:34.352 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:25:34.347+0000 7f627d422080 0 bucket checkpoint complete 2021-02-11T06:25:34.379 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:34.379 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:34.380 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-24 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:25:34.430 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:25:34.430 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:25:34.669 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:25:34.662+0000 7f9440dbf080 1 bucket sync caught up with source: 2021-02-11T06:25:34.670 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , , , , 00000000001.25.12, , ] 2021-02-11T06:25:34.670 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , , , , 00000000001.25.12, , ] 2021-02-11T06:25:34.670 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:25:34.662+0000 7f9440dbf080 0 bucket checkpoint complete 2021-02-11T06:25:34.702 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-24 zones={test-zone2, test-zone1} 2021-02-11T06:25:34.712 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:25:34.759 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:25:34.759 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:25:34.766 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:25:34.821 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:25:34.826 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-24 zones={test-zone2, test-zone1} 2021-02-11T06:25:34.827 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-24 zones={test-zone3, test-zone1} 2021-02-11T06:25:34.835 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:25:34.883 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:25:34.883 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:25:34.889 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:25:34.946 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:25:34.951 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-24 zones={test-zone3, test-zone1} 2021-02-11T06:25:34.951 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-24 zones={test-zone3, test-zone2} 2021-02-11T06:25:34.959 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:25:35.006 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:25:35.007 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:25:35.012 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:25:35.082 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:25:35.087 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-24 zones={test-zone3, test-zone2} 2021-02-11T06:25:35.087 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:35.087 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:35.088 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-25 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:25:35.135 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:25:35.136 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:25:35.377 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:25:35.370+0000 7f2e8adde080 1 bucket sync caught up with source: 2021-02-11T06:25:35.377 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , , , , 00000000001.11.12, , ] 2021-02-11T06:25:35.377 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , , , , 00000000001.11.12, , ] 2021-02-11T06:25:35.377 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:25:35.370+0000 7f2e8adde080 0 bucket checkpoint complete 2021-02-11T06:25:35.403 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:35.403 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:35.403 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-25 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:25:35.454 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:25:35.454 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:25:35.693 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:25:35.691+0000 7fd8010fe080 1 bucket sync caught up with source: 2021-02-11T06:25:35.693 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, , , , , , , , 00000000001.11.12, , ] 2021-02-11T06:25:35.693 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, , , , , , , , 00000000001.11.12, , ] 2021-02-11T06:25:35.693 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:25:35.691+0000 7fd8010fe080 0 bucket checkpoint complete 2021-02-11T06:25:35.720 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:35.720 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:35.720 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-25 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:25:35.768 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:25:35.768 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:25:36.014 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:25:36.010+0000 7f0fa2de1080 1 bucket sync caught up with source: 2021-02-11T06:25:36.014 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , , , , 00000000001.11.12, , ] 2021-02-11T06:25:36.014 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , , , , 00000000001.11.12, , ] 2021-02-11T06:25:36.014 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:25:36.010+0000 7f0fa2de1080 0 bucket checkpoint complete 2021-02-11T06:25:36.043 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:36.043 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:36.043 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-25 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:25:36.091 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:25:36.091 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:25:36.349 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:25:36.347+0000 7f57c57a8080 1 bucket sync caught up with source: 2021-02-11T06:25:36.349 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, , , , , , , , 00000000003.26.6, , ] 2021-02-11T06:25:36.349 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, , , , , , , , 00000000003.26.6, , ] 2021-02-11T06:25:36.350 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:25:36.347+0000 7f57c57a8080 0 bucket checkpoint complete 2021-02-11T06:25:36.380 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:36.380 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:36.380 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-25 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:25:36.428 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:25:36.428 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:25:36.692 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:25:36.687+0000 7fa048e65080 1 bucket sync caught up with source: 2021-02-11T06:25:36.692 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, , , , , , , , 00000000003.26.6, , ] 2021-02-11T06:25:36.692 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, , , , , , , , 00000000003.26.6, , ] 2021-02-11T06:25:36.692 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:25:36.687+0000 7fa048e65080 0 bucket checkpoint complete 2021-02-11T06:25:36.724 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:36.724 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:36.724 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-25 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:25:36.770 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:25:36.770 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:25:37.021 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:25:37.019+0000 7f94a4c11080 1 bucket sync caught up with source: 2021-02-11T06:25:37.021 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, , , , , , , , 00000000001.21.12, , ] 2021-02-11T06:25:37.021 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, , , , , , , , 00000000001.21.12, , ] 2021-02-11T06:25:37.022 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:25:37.019+0000 7f94a4c11080 0 bucket checkpoint complete 2021-02-11T06:25:37.050 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:37.050 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:37.050 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-25 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:25:37.098 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:25:37.098 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:25:37.350 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:25:37.346+0000 7f5e739d4080 1 bucket sync caught up with source: 2021-02-11T06:25:37.350 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , , , , 00000000001.21.12, , ] 2021-02-11T06:25:37.350 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , , , , 00000000001.21.12, , ] 2021-02-11T06:25:37.350 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:25:37.346+0000 7f5e739d4080 0 bucket checkpoint complete 2021-02-11T06:25:37.380 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-25 zones={test-zone2, test-zone1} 2021-02-11T06:25:37.390 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:25:37.439 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:25:37.439 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:25:37.446 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:25:37.505 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:25:37.511 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-25 zones={test-zone2, test-zone1} 2021-02-11T06:25:37.511 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-25 zones={test-zone3, test-zone1} 2021-02-11T06:25:37.519 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:25:37.566 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:25:37.567 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:25:37.572 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:25:37.629 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:25:37.635 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-25 zones={test-zone3, test-zone1} 2021-02-11T06:25:37.635 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-25 zones={test-zone3, test-zone2} 2021-02-11T06:25:37.644 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:25:37.690 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:25:37.691 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:25:37.697 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:25:37.752 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:25:37.757 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-25 zones={test-zone3, test-zone2} 2021-02-11T06:25:37.758 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:37.758 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:37.758 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-26 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:25:37.801 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:25:37.801 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:25:38.047 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:25:38.042+0000 7fe3f44c7080 1 bucket sync caught up with source: 2021-02-11T06:25:38.047 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , , , , 00000000001.13.12, , ] 2021-02-11T06:25:38.047 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , , , , 00000000001.13.12, , ] 2021-02-11T06:25:38.047 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:25:38.042+0000 7fe3f44c7080 0 bucket checkpoint complete 2021-02-11T06:25:38.078 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:38.078 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:38.079 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-26 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:25:38.126 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:25:38.127 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:25:38.372 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:25:38.367+0000 7ffba4731080 1 bucket sync caught up with source: 2021-02-11T06:25:38.372 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, , , , , , , , 00000000001.13.12, , ] 2021-02-11T06:25:38.372 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, , , , , , , , 00000000001.13.12, , ] 2021-02-11T06:25:38.372 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:25:38.367+0000 7ffba4731080 0 bucket checkpoint complete 2021-02-11T06:25:38.403 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:38.403 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:38.403 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-26 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:25:38.452 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:25:38.452 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:25:38.686 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:25:38.682+0000 7f10c0181080 1 bucket sync caught up with source: 2021-02-11T06:25:38.686 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , , , , 00000000001.13.12, , ] 2021-02-11T06:25:38.686 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , , , , 00000000001.13.12, , ] 2021-02-11T06:25:38.686 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:25:38.682+0000 7f10c0181080 0 bucket checkpoint complete 2021-02-11T06:25:38.716 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:38.716 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:38.716 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-26 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:25:38.764 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:25:38.765 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:25:39.010 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:25:39.007+0000 7f68eafff080 1 bucket sync caught up with source: 2021-02-11T06:25:39.010 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, , , , , , , , 00000000001.17.12, , ] 2021-02-11T06:25:39.010 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, , , , , , , , 00000000001.17.12, , ] 2021-02-11T06:25:39.010 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:25:39.007+0000 7f68eafff080 0 bucket checkpoint complete 2021-02-11T06:25:39.040 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:39.040 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:39.041 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-26 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:25:39.091 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:25:39.092 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:25:39.344 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:25:39.339+0000 7f577e175080 1 bucket sync caught up with source: 2021-02-11T06:25:39.344 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, , , , , , , , 00000000001.17.12, , ] 2021-02-11T06:25:39.344 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, , , , , , , , 00000000001.17.12, , ] 2021-02-11T06:25:39.345 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:25:39.339+0000 7f577e175080 0 bucket checkpoint complete 2021-02-11T06:25:39.378 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:39.378 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:39.379 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-26 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:25:39.425 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:25:39.425 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:25:39.653 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:25:39.651+0000 7f69a0c5a080 1 bucket sync caught up with source: 2021-02-11T06:25:39.653 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, , , , , , , , 00000000003.23.6, , ] 2021-02-11T06:25:39.653 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, , , , , , , , 00000000003.23.6, , ] 2021-02-11T06:25:39.653 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:25:39.651+0000 7f69a0c5a080 0 bucket checkpoint complete 2021-02-11T06:25:39.685 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:39.685 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:25:39.685 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-26 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:25:39.732 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:25:39.732 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:25:39.969 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:25:39.962+0000 7f3c83e56080 1 bucket sync caught up with source: 2021-02-11T06:25:39.970 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , , , , 00000000003.23.6, , ] 2021-02-11T06:25:39.970 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , , , , 00000000003.23.6, , ] 2021-02-11T06:25:39.970 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:25:39.962+0000 7f3c83e56080 0 bucket checkpoint complete 2021-02-11T06:25:40.001 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-26 zones={test-zone2, test-zone1} 2021-02-11T06:25:40.011 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:25:40.059 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:25:40.059 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:25:40.066 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:25:40.121 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:25:40.126 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-26 zones={test-zone2, test-zone1} 2021-02-11T06:25:40.126 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-26 zones={test-zone3, test-zone1} 2021-02-11T06:25:40.133 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:25:40.182 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:25:40.183 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:25:40.189 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:25:40.244 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:25:40.249 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-26 zones={test-zone3, test-zone1} 2021-02-11T06:25:40.249 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-26 zones={test-zone3, test-zone2} 2021-02-11T06:25:40.257 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:25:40.302 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:25:40.303 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:25:40.308 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T06:25:40.360 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T06:25:40.365 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-26 zones={test-zone3, test-zone2} 2021-02-11T06:25:40.365 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_suspended_delete_marker_full_sync ... ok 2021-02-11T06:25:40.366 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xscaju-27 2021-02-11T06:25:40.405 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xscaju-28 2021-02-11T06:25:40.471 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xscaju-29 2021-02-11T06:25:40.714 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_versioning ... ok 2021-02-11T06:25:40.715 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xscaju-30 2021-02-11T06:25:40.752 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xscaju-31 2021-02-11T06:25:40.816 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xscaju-32 2021-02-11T06:25:41.074 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_acl ... ok 2021-02-11T06:25:41.075 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xscaju-33 2021-02-11T06:25:41.113 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xscaju-34 2021-02-11T06:25:41.170 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xscaju-35 2021-02-11T06:25:43.427 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_cors ... ok 2021-02-11T06:25:43.427 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xscaju-36 2021-02-11T06:25:43.466 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xscaju-37 2021-02-11T06:25:43.523 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xscaju-38 2021-02-11T06:25:43.584 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:25:43.584 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:25:43.585 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:25:43.622 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:25:43.622 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:25:43.853 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=7c4a812f-81e4-4a72-acce-0ee18589648c 2021-02-11T06:25:43.897 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613024743.556753_312.1","last_update":"2021-02-11T06:25:43.556753Z"},{"marker":"1_1613024301.142235_74.1","last_update":"2021-02-11T06:18:21.142235Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:25:43.902 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T06:25:43.902 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:25:43.902 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:25:43.903 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:25:43.943 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:25:43.943 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:25:44.208 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7c4a812f-81e4-4a72-acce-0ee18589648c","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613024743.556753_312.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:25:43.556753Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:25:49.357 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T06:25:49.358 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T06:25:49.358 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:25:49.358 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:25:49.358 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:25:49.400 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:25:49.400 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:25:49.660 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7c4a812f-81e4-4a72-acce-0ee18589648c","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613024743.556753_312.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:25:43.556753Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:25:49.661 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T06:25:49.661 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:25:49.661 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:25:49.661 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:25:49.661 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:25:49.705 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:25:49.706 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:25:49.968 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7c4a812f-81e4-4a72-acce-0ee18589648c","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613024743.556753_312.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:25:43.556753Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:25:49.969 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:25:50.334 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_delete_notempty ... ok 2021-02-11T06:25:50.335 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xscaju-39 2021-02-11T06:25:50.378 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xscaju-40 2021-02-11T06:25:50.469 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xscaju-41 2021-02-11T06:25:50.535 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:25:50.535 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:25:50.536 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:25:50.570 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:25:50.570 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:25:50.782 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=7c4a812f-81e4-4a72-acce-0ee18589648c 2021-02-11T06:25:50.812 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613024750.509787_328.1","last_update":"2021-02-11T06:25:50.509787Z"},{"marker":"1_1613024301.142235_74.1","last_update":"2021-02-11T06:18:21.142235Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:25:50.812 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T06:25:50.813 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:25:50.813 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:25:50.813 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:25:50.853 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:25:50.854 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:25:51.109 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7c4a812f-81e4-4a72-acce-0ee18589648c","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613024750.509787_328.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:25:50.509787Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:25:51.110 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T06:25:51.110 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T06:25:51.110 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:25:51.110 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:25:51.110 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:25:51.151 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:25:51.151 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:25:51.416 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7c4a812f-81e4-4a72-acce-0ee18589648c","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613024750.509787_328.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:25:50.509787Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:25:51.416 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T06:25:51.416 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:25:51.416 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:25:51.416 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:25:51.417 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:25:51.456 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:25:51.456 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:25:51.711 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7c4a812f-81e4-4a72-acce-0ee18589648c","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613024750.509787_328.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:25:50.509787Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:25:51.712 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:25:51.712 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2021-02-11T06:25:51.712 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T06:25:51.713 INFO:tasks.rgw.c1.client.1.smithi061.stdout:2021-02-11T06:25:51.711+0000 7ff3d2ffd700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 19941) UID: 0 2021-02-11T06:25:51.713 INFO:tasks.rgw.c1.client.1.smithi061.stdout:2021-02-11T06:25:51.711+0000 7ff3f6cea0c0 -1 shutting down 2021-02-11T06:25:55.671 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2021-02-11T06:25:57.813 INFO:tasks.rgw.c1.client.1:Stopped 2021-02-11T06:25:57.814 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:25:57.814 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:25:57.814 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --master --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T06:25:57.852 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:25:57.853 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:25:57.882 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:25:57.877+0000 7fb109fa2080 0 NOTICE: overriding master zone: 5f91a5b1-d54f-4a3d-a100-b2911a08a2e7 2021-02-11T06:25:57.883 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:25:57.877+0000 7fb109fa2080 0 failed reading obj info from .rgw.root:zone_info.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7: (2) No such file or directory 2021-02-11T06:25:57.883 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:25:57.877+0000 7fb109fa2080 0 WARNING: could not read zone params for zone id=5f91a5b1-d54f-4a3d-a100-b2911a08a2e7 name=test-zone1 2021-02-11T06:25:57.883 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:25:57.877+0000 7fb109fa2080 0 failed reading obj info from .rgw.root:zone_info.bca56477-0808-4ebe-8649-b1169fc76383: (2) No such file or directory 2021-02-11T06:25:57.883 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:25:57.877+0000 7fb109fa2080 0 WARNING: could not read zone params for zone id=bca56477-0808-4ebe-8649-b1169fc76383 name=test-zone3 2021-02-11T06:25:57.897 INFO:teuthology.orchestra.run.smithi172.stdout:{"id":"e4b3955d-973b-4dd2-ae54-b7e681fbeb21","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"307AEWIWKOCPYEOH","secret_key":"qEUy5RJVuZgF4tphJ6ujQNFJaaIFkUpT"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"a797f55e-5ffc-4df8-acfc-1a39dd991822","notif_pool":"test-zone2.rgw.log:notif"} 2021-02-11T06:25:57.897 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:25:57.897 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:25:57.897 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T06:25:57.979 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:25:57.979 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:25:58.300 INFO:teuthology.orchestra.run.smithi172.stdout:{"id":"2afe6ddd-1cf8-433a-ae51-67de25931f7b","epoch":1,"predecessor_uuid":"7c4a812f-81e4-4a72-acce-0ee18589648c","sync_status":["1_1613024750.509787_328.1","1_1613024301.142235_74.1","",""],"period_map":{"id":"2afe6ddd-1cf8-433a-ae51-67de25931f7b","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi061.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"e4b3955d-973b-4dd2-ae54-b7e681fbeb21","zones":[{"id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7","name":"test-zone1","endpoints":["http://smithi061.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"bca56477-0808-4ebe-8649-b1169fc76383","name":"test-zone3","endpoints":["http://smithi061.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"d7ed052a-b089-4bac-932c-80450ae3dd3d","name":"test-zone4","endpoints":["http://smithi172.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"e4b3955d-973b-4dd2-ae54-b7e681fbeb21","name":"test-zone2","endpoints":["http://smithi172.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"a797f55e-5ffc-4df8-acfc-1a39dd991822","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7","val":3877615972},{"key":"bca56477-0808-4ebe-8649-b1169fc76383","val":2043214290},{"key":"d7ed052a-b089-4bac-932c-80450ae3dd3d","val":3210761346},{"key":"e4b3955d-973b-4dd2-ae54-b7e681fbeb21","val":3820998694}]},"master_zonegroup":"test-zonegroup","master_zone":"e4b3955d-973b-4dd2-ae54-b7e681fbeb21","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"a797f55e-5ffc-4df8-acfc-1a39dd991822","realm_name":"test-realm","realm_epoch":3} 2021-02-11T06:25:58.301 INFO:rgw_multi.tests:Set master zone=test-zone2, waiting 60s for reconfiguration.. 2021-02-11T06:26:01.278 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2021-02-11T06:26:58.302 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xscaju-42 2021-02-11T06:26:58.381 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xscaju-43 2021-02-11T06:26:58.427 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:26:58.428 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:26:58.428 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:26:58.464 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:26:58.464 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:26:58.690 INFO:teuthology.orchestra.run.smithi172.stderr:No --period given, using current period=2afe6ddd-1cf8-433a-ae51-67de25931f7b 2021-02-11T06:26:58.720 INFO:teuthology.orchestra.run.smithi172.stdout:[{"marker":"1_1613024818.417656_271.1","last_update":"2021-02-11T06:26:58.417656Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:26:58.721 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone1 2021-02-11T06:26:58.721 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:26:58.721 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:26:58.721 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:26:58.763 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:26:58.763 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:26:59.010 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"2afe6ddd-1cf8-433a-ae51-67de25931f7b","realm_epoch":3},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613024818.417656_271.1","next_step_marker":"","total_entries":80,"pos":0,"timestamp":"2021-02-11T06:26:58.417656Z","realm_epoch":3}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":3}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":3}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":3}}]},"full_sync":{"total":80,"complete":80}} 2021-02-11T06:26:59.010 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone1 2021-02-11T06:26:59.011 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:26:59.011 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:26:59.011 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --master --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T06:26:59.092 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:26:59.093 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:26:59.119 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:26:59.114+0000 7f2f3d11e080 0 NOTICE: overriding master zone: e4b3955d-973b-4dd2-ae54-b7e681fbeb21 2021-02-11T06:26:59.121 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:26:59.118+0000 7f2f3d11e080 0 failed reading obj info from .rgw.root:zone_info.d7ed052a-b089-4bac-932c-80450ae3dd3d: (2) No such file or directory 2021-02-11T06:26:59.121 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:26:59.118+0000 7f2f3d11e080 0 WARNING: could not read zone params for zone id=d7ed052a-b089-4bac-932c-80450ae3dd3d name=test-zone4 2021-02-11T06:26:59.121 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:26:59.118+0000 7f2f3d11e080 0 failed reading obj info from .rgw.root:zone_info.e4b3955d-973b-4dd2-ae54-b7e681fbeb21: (2) No such file or directory 2021-02-11T06:26:59.121 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:26:59.118+0000 7f2f3d11e080 0 WARNING: could not read zone params for zone id=e4b3955d-973b-4dd2-ae54-b7e681fbeb21 name=test-zone2 2021-02-11T06:26:59.131 INFO:teuthology.orchestra.run.smithi061.stdout:{"id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7","name":"test-zone1","domain_root":"test-zone1.rgw.meta:root","control_pool":"test-zone1.rgw.control","gc_pool":"test-zone1.rgw.log:gc","lc_pool":"test-zone1.rgw.log:lc","log_pool":"test-zone1.rgw.log","intent_log_pool":"test-zone1.rgw.log:intent","usage_log_pool":"test-zone1.rgw.log:usage","roles_pool":"test-zone1.rgw.meta:roles","reshard_pool":"test-zone1.rgw.log:reshard","user_keys_pool":"test-zone1.rgw.meta:users.keys","user_email_pool":"test-zone1.rgw.meta:users.email","user_swift_pool":"test-zone1.rgw.meta:users.swift","user_uid_pool":"test-zone1.rgw.meta:users.uid","otp_pool":"test-zone1.rgw.otp","system_key":{"access_key":"307AEWIWKOCPYEOH","secret_key":"qEUy5RJVuZgF4tphJ6ujQNFJaaIFkUpT"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone1.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone1.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone1.rgw.buckets.non-ec","index_type":0}}],"realm_id":"a797f55e-5ffc-4df8-acfc-1a39dd991822","notif_pool":"test-zone1.rgw.log:notif"} 2021-02-11T06:26:59.132 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:26:59.132 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:26:59.132 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T06:26:59.212 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:26:59.212 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:26:59.550 INFO:teuthology.orchestra.run.smithi061.stdout:{"id":"0d061ff0-ede8-4baa-beef-d975d317d505","epoch":1,"predecessor_uuid":"2afe6ddd-1cf8-433a-ae51-67de25931f7b","sync_status":["1_1613024818.417656_271.1","","",""],"period_map":{"id":"0d061ff0-ede8-4baa-beef-d975d317d505","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi061.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7","zones":[{"id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7","name":"test-zone1","endpoints":["http://smithi061.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"bca56477-0808-4ebe-8649-b1169fc76383","name":"test-zone3","endpoints":["http://smithi061.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"d7ed052a-b089-4bac-932c-80450ae3dd3d","name":"test-zone4","endpoints":["http://smithi172.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"e4b3955d-973b-4dd2-ae54-b7e681fbeb21","name":"test-zone2","endpoints":["http://smithi172.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"a797f55e-5ffc-4df8-acfc-1a39dd991822","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7","val":3877615972},{"key":"bca56477-0808-4ebe-8649-b1169fc76383","val":2043214290},{"key":"d7ed052a-b089-4bac-932c-80450ae3dd3d","val":3210761346},{"key":"e4b3955d-973b-4dd2-ae54-b7e681fbeb21","val":3820998694}]},"master_zonegroup":"test-zonegroup","master_zone":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"a797f55e-5ffc-4df8-acfc-1a39dd991822","realm_name":"test-realm","realm_epoch":4} 2021-02-11T06:26:59.551 INFO:rgw_multi.tests:Set master zone=test-zone1, waiting 60s for reconfiguration.. 2021-02-11T06:27:59.552 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xscaju-44 2021-02-11T06:27:59.605 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xscaju-45 2021-02-11T06:27:59.682 INFO:tasks.rgw.c1.client.1:Restarting daemon 2021-02-11T06:27:59.682 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2021-02-11T06:27:59.685 INFO:tasks.rgw.c1.client.1:Started 2021-02-11T06:27:59.685 DEBUG:teuthology.orchestra.run.smithi061:> curl http://smithi061.front.sepia.ceph.com:8001 2021-02-11T06:27:59.699 INFO:teuthology.orchestra.run.smithi061.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-11T06:27:59.699 INFO:teuthology.orchestra.run.smithi061.stderr: Dload Upload Total Spent Left Speed 2021-02-11T06:27:59.703 INFO:teuthology.orchestra.run.smithi061.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi061.front.sepia.ceph.com port 8001: Connection refused 2021-02-11T06:27:59.704 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-02-11T06:28:00.705 DEBUG:teuthology.orchestra.run.smithi061:> curl http://smithi061.front.sepia.ceph.com:8001 2021-02-11T06:28:07.634 INFO:teuthology.orchestra.run.smithi061.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-11T06:28:07.634 INFO:teuthology.orchestra.run.smithi061.stderr: Dload Upload Total Spent Left Speed 2021-02-11T06:28:07.643 INFO:teuthology.orchestra.run.smithi061.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 26750 0 --:--:-- --:--:-- --:--:-- 26750 2021-02-11T06:28:07.644 INFO:teuthology.orchestra.run.smithi061.stdout:anonymous 2021-02-11T06:28:07.644 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:28:07.644 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:28:07.645 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:28:07.726 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:28:07.726 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:28:07.957 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=0d061ff0-ede8-4baa-beef-d975d317d505 2021-02-11T06:28:30.561 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613024879.661926_95.1","last_update":"2021-02-11T06:27:59.661926Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:28:30.562 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T06:28:30.562 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:28:30.563 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:28:30.563 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:28:30.609 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:28:30.610 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:28:30.869 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613024879.661926_95.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:27:59.661926Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:28:30.869 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T06:28:30.869 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T06:28:30.869 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:28:30.869 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:28:30.870 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:28:30.910 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:28:30.910 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:28:31.166 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613024879.661926_95.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:27:59.661926Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:28:31.167 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T06:28:31.167 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:28:31.167 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:28:31.167 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:28:31.167 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:28:31.209 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:28:31.209 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:28:31.469 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613024879.661926_95.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:27:59.661926Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:28:31.469 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:28:31.469 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-39 zones={test-zone2, test-zone1} 2021-02-11T06:28:31.490 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:28:31.539 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:28:31.605 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-39 zones={test-zone2, test-zone1} 2021-02-11T06:28:31.605 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-39 zones={test-zone3, test-zone1} 2021-02-11T06:28:31.622 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:28:31.671 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:28:31.734 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-39 zones={test-zone3, test-zone1} 2021-02-11T06:28:31.734 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-39 zones={test-zone1, test-zone2} 2021-02-11T06:28:31.745 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:28:31.790 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:28:31.855 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-39 zones={test-zone1, test-zone2} 2021-02-11T06:28:31.855 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-39 zones={test-zone3, test-zone2} 2021-02-11T06:28:31.866 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:28:31.915 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:28:31.981 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-39 zones={test-zone3, test-zone2} 2021-02-11T06:28:31.982 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-39 zones={test-zone1, test-zone3} 2021-02-11T06:28:31.995 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:28:32.043 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:28:32.108 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-39 zones={test-zone1, test-zone3} 2021-02-11T06:28:32.108 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-39 zones={test-zone2, test-zone3} 2021-02-11T06:28:32.118 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:28:32.166 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:28:32.230 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-39 zones={test-zone2, test-zone3} 2021-02-11T06:28:32.230 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-40 zones={test-zone2, test-zone1} 2021-02-11T06:28:32.246 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:28:32.294 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:28:32.364 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-40 zones={test-zone2, test-zone1} 2021-02-11T06:28:32.364 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-40 zones={test-zone3, test-zone1} 2021-02-11T06:28:32.376 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:28:32.422 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:28:32.485 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-40 zones={test-zone3, test-zone1} 2021-02-11T06:28:32.485 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-40 zones={test-zone1, test-zone2} 2021-02-11T06:28:32.496 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:28:32.543 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:28:32.615 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-40 zones={test-zone1, test-zone2} 2021-02-11T06:28:33.425 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-40 zones={test-zone3, test-zone2} 2021-02-11T06:28:33.438 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:28:33.487 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:28:33.549 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-40 zones={test-zone3, test-zone2} 2021-02-11T06:28:33.549 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-40 zones={test-zone1, test-zone3} 2021-02-11T06:28:33.560 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:28:33.606 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:28:33.669 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-40 zones={test-zone1, test-zone3} 2021-02-11T06:28:33.669 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-40 zones={test-zone2, test-zone3} 2021-02-11T06:28:33.679 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:28:33.726 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:28:33.789 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-40 zones={test-zone2, test-zone3} 2021-02-11T06:28:33.790 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-41 zones={test-zone2, test-zone1} 2021-02-11T06:28:33.804 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:28:33.854 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:28:33.917 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-41 zones={test-zone2, test-zone1} 2021-02-11T06:28:33.917 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-41 zones={test-zone3, test-zone1} 2021-02-11T06:28:33.929 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:28:33.974 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:28:34.037 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-41 zones={test-zone3, test-zone1} 2021-02-11T06:28:34.037 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-41 zones={test-zone1, test-zone2} 2021-02-11T06:28:34.048 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:28:34.094 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:28:34.157 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-41 zones={test-zone1, test-zone2} 2021-02-11T06:28:34.157 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-41 zones={test-zone3, test-zone2} 2021-02-11T06:28:34.167 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:28:34.214 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:28:34.277 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-41 zones={test-zone3, test-zone2} 2021-02-11T06:28:34.277 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-41 zones={test-zone1, test-zone3} 2021-02-11T06:28:34.288 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:28:34.334 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:28:34.395 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-41 zones={test-zone1, test-zone3} 2021-02-11T06:28:34.395 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-41 zones={test-zone2, test-zone3} 2021-02-11T06:28:34.409 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:28:34.461 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:28:34.545 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-41 zones={test-zone2, test-zone3} 2021-02-11T06:28:34.545 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-42 zones={test-zone2, test-zone1} 2021-02-11T06:28:34.562 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:28:34.611 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:28:34.691 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-42 zones={test-zone2, test-zone1} 2021-02-11T06:28:34.691 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-42 zones={test-zone3, test-zone1} 2021-02-11T06:28:34.704 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:28:34.757 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:28:34.833 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-42 zones={test-zone3, test-zone1} 2021-02-11T06:28:34.833 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-42 zones={test-zone1, test-zone2} 2021-02-11T06:28:34.849 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:28:34.906 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:28:34.985 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-42 zones={test-zone1, test-zone2} 2021-02-11T06:28:34.985 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-42 zones={test-zone3, test-zone2} 2021-02-11T06:28:34.994 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:28:35.044 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:28:35.117 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-42 zones={test-zone3, test-zone2} 2021-02-11T06:28:35.117 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-42 zones={test-zone1, test-zone3} 2021-02-11T06:28:35.126 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:28:35.175 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:28:35.241 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-42 zones={test-zone1, test-zone3} 2021-02-11T06:28:35.242 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-42 zones={test-zone2, test-zone3} 2021-02-11T06:28:35.250 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:28:35.299 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:28:35.362 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-42 zones={test-zone2, test-zone3} 2021-02-11T06:28:35.362 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-43 zones={test-zone2, test-zone1} 2021-02-11T06:28:35.371 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:28:35.418 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:28:35.481 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-43 zones={test-zone2, test-zone1} 2021-02-11T06:28:35.482 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-43 zones={test-zone3, test-zone1} 2021-02-11T06:28:35.492 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:28:35.539 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:28:35.600 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-43 zones={test-zone3, test-zone1} 2021-02-11T06:28:35.600 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-43 zones={test-zone1, test-zone2} 2021-02-11T06:28:35.611 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:28:35.662 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:28:35.725 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-43 zones={test-zone1, test-zone2} 2021-02-11T06:28:35.726 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-43 zones={test-zone3, test-zone2} 2021-02-11T06:28:35.736 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:28:35.782 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:28:35.845 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-43 zones={test-zone3, test-zone2} 2021-02-11T06:28:35.845 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-43 zones={test-zone1, test-zone3} 2021-02-11T06:28:35.857 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:28:35.906 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:28:35.968 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-43 zones={test-zone1, test-zone3} 2021-02-11T06:28:35.969 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-43 zones={test-zone2, test-zone3} 2021-02-11T06:28:35.979 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:28:36.027 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:28:36.091 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-43 zones={test-zone2, test-zone3} 2021-02-11T06:28:36.091 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-44 zones={test-zone2, test-zone1} 2021-02-11T06:28:36.100 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:28:36.146 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:28:36.206 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-44 zones={test-zone2, test-zone1} 2021-02-11T06:28:36.206 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-44 zones={test-zone3, test-zone1} 2021-02-11T06:28:36.219 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:28:36.267 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:28:36.328 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-44 zones={test-zone3, test-zone1} 2021-02-11T06:28:36.328 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-44 zones={test-zone1, test-zone2} 2021-02-11T06:28:36.337 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:28:36.382 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:28:36.445 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-44 zones={test-zone1, test-zone2} 2021-02-11T06:28:36.445 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-44 zones={test-zone3, test-zone2} 2021-02-11T06:28:36.455 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:28:36.507 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:28:36.569 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-44 zones={test-zone3, test-zone2} 2021-02-11T06:28:36.570 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-44 zones={test-zone1, test-zone3} 2021-02-11T06:28:36.582 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:28:36.630 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:28:36.688 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-44 zones={test-zone1, test-zone3} 2021-02-11T06:28:36.688 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-44 zones={test-zone2, test-zone3} 2021-02-11T06:28:36.698 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:28:36.746 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:28:36.809 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-44 zones={test-zone2, test-zone3} 2021-02-11T06:28:36.809 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-45 zones={test-zone2, test-zone1} 2021-02-11T06:28:36.818 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:28:36.870 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:28:36.934 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-45 zones={test-zone2, test-zone1} 2021-02-11T06:28:36.934 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-45 zones={test-zone3, test-zone1} 2021-02-11T06:28:36.943 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:28:36.994 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:28:37.056 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-45 zones={test-zone3, test-zone1} 2021-02-11T06:28:37.056 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-45 zones={test-zone1, test-zone2} 2021-02-11T06:28:37.066 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:28:37.111 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:28:37.177 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-45 zones={test-zone1, test-zone2} 2021-02-11T06:28:37.177 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-45 zones={test-zone3, test-zone2} 2021-02-11T06:28:37.185 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:28:37.230 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:28:37.291 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-45 zones={test-zone3, test-zone2} 2021-02-11T06:28:37.291 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-45 zones={test-zone1, test-zone3} 2021-02-11T06:28:37.303 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:28:37.351 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:28:37.408 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-45 zones={test-zone1, test-zone3} 2021-02-11T06:28:37.408 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-45 zones={test-zone2, test-zone3} 2021-02-11T06:28:37.418 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:28:37.466 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:28:37.529 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-45 zones={test-zone2, test-zone3} 2021-02-11T06:28:37.530 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '7c4a812f-81e4-4a72-acce-0ee18589648c', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:28:37.530 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '7c4a812f-81e4-4a72-acce-0ee18589648c', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:28:37.530 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 7c4a812f-81e4-4a72-acce-0ee18589648c --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:28:37.565 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:28:37.565 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:28:37.847 INFO:teuthology.orchestra.run.smithi061.stdout:[{"id":"1_1613024135.173475_43.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2021-02-11T06:15:35.173475237Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SbNMO4uv9J_5-KcMlPHXiJf","ver":1},"status":{"status":"write"}}},{"id":"1_1613024135.181851_48.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2021-02-11T06:15:35.181851493Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SbNMO4uv9J_5-KcMlPHXiJf","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024304.838286_55.1","section":"bucket.instance","name":"xscaju-10:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.10","timestamp":"2021-02-11T06:18:24.838286662Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vUDtGJ3GBQdG6kVLldnqDId","ver":1},"status":{"status":"write"}}},{"id":"1_1613024304.844317_62.1","section":"bucket.instance","name":"xscaju-10:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.10","timestamp":"2021-02-11T06:18:24.844317783Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vUDtGJ3GBQdG6kVLldnqDId","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024304.846426_63.1","section":"bucket","name":"xscaju-10","timestamp":"2021-02-11T06:18:24.846426466Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pfKfSMc5TvrJjKRiccmEVID","ver":1},"status":{"status":"write"}}},{"id":"1_1613024304.852830_64.1","section":"bucket","name":"xscaju-10","timestamp":"2021-02-11T06:18:24.852830013Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pfKfSMc5TvrJjKRiccmEVID","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024304.876275_65.1","section":"bucket.instance","name":"xscaju-11:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.11","timestamp":"2021-02-11T06:18:24.876275792Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7O37Ypf5_pArN3rvyGMahZQ","ver":1},"status":{"status":"write"}}},{"id":"1_1613024304.882604_66.1","section":"bucket.instance","name":"xscaju-11:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.11","timestamp":"2021-02-11T06:18:24.882604435Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7O37Ypf5_pArN3rvyGMahZQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024304.884842_67.1","section":"bucket","name":"xscaju-11","timestamp":"2021-02-11T06:18:24.884842370Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hC41ht1E97pbsgCJUm_8lj_","ver":1},"status":{"status":"write"}}},{"id":"1_1613024304.890580_68.1","section":"bucket","name":"xscaju-11","timestamp":"2021-02-11T06:18:24.890580142Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hC41ht1E97pbsgCJUm_8lj_","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024304.936648_69.1","section":"bucket.instance","name":"xscaju-12:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.12","timestamp":"2021-02-11T06:18:24.936648345Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rdqYRx6VnHnoYGod-A0qeUL","ver":1},"status":{"status":"write"}}},{"id":"1_1613024304.943318_70.1","section":"bucket.instance","name":"xscaju-12:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.12","timestamp":"2021-02-11T06:18:24.943318666Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rdqYRx6VnHnoYGod-A0qeUL","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024304.946295_71.1","section":"bucket","name":"xscaju-12","timestamp":"2021-02-11T06:18:24.946295639Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YM8ZNKbVwbAQ1oc9Cz4_OM_","ver":1},"status":{"status":"write"}}},{"id":"1_1613024304.951770_72.1","section":"bucket","name":"xscaju-12","timestamp":"2021-02-11T06:18:24.951770048Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YM8ZNKbVwbAQ1oc9Cz4_OM_","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024341.501587_78.1","section":"bucket.instance","name":"xscaju-13:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.13","timestamp":"2021-02-11T06:19:01.501587544Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BCFlY5FQTgcsjiMtHT5pskt","ver":1},"status":{"status":"write"}}},{"id":"1_1613024341.507190_81.1","section":"bucket.instance","name":"xscaju-13:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.13","timestamp":"2021-02-11T06:19:01.507190496Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BCFlY5FQTgcsjiMtHT5pskt","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024341.509363_82.1","section":"bucket","name":"xscaju-13","timestamp":"2021-02-11T06:19:01.509363808Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_i2XT5Un3mY87cqRlmnmKTE1","ver":1},"status":{"status":"write"}}},{"id":"1_1613024341.515079_83.1","section":"bucket","name":"xscaju-13","timestamp":"2021-02-11T06:19:01.515079048Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_i2XT5Un3mY87cqRlmnmKTE1","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024341.540023_84.1","section":"bucket.instance","name":"xscaju-14:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.14","timestamp":"2021-02-11T06:19:01.540023533Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OG9OQa6kjk0iKLLVPQDaiG-","ver":1},"status":{"status":"write"}}},{"id":"1_1613024341.546055_85.1","section":"bucket.instance","name":"xscaju-14:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.14","timestamp":"2021-02-11T06:19:01.546055113Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OG9OQa6kjk0iKLLVPQDaiG-","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024341.548182_86.1","section":"bucket","name":"xscaju-14","timestamp":"2021-02-11T06:19:01.548182976Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LolNl85pkAeXvxWPRnXMOqS","ver":1},"status":{"status":"write"}}},{"id":"1_1613024341.553747_87.1","section":"bucket","name":"xscaju-14","timestamp":"2021-02-11T06:19:01.553747682Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LolNl85pkAeXvxWPRnXMOqS","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024341.605553_88.1","section":"bucket.instance","name":"xscaju-15:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.15","timestamp":"2021-02-11T06:19:01.605553336Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UrhgCwLCqd88P1DuEAbHlw5","ver":1},"status":{"status":"write"}}},{"id":"1_1613024341.611354_89.1","section":"bucket.instance","name":"xscaju-15:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.15","timestamp":"2021-02-11T06:19:01.611354865Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UrhgCwLCqd88P1DuEAbHlw5","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024341.613945_90.1","section":"bucket","name":"xscaju-15","timestamp":"2021-02-11T06:19:01.613945279Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FYxrvNRyhtUMgtpEdrH0XpH","ver":1},"status":{"status":"write"}}},{"id":"1_1613024341.620645_91.1","section":"bucket","name":"xscaju-15","timestamp":"2021-02-11T06:19:01.620645045Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FYxrvNRyhtUMgtpEdrH0XpH","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024373.488563_97.1","section":"bucket.instance","name":"xscaju-16:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.16","timestamp":"2021-02-11T06:19:33.488563051Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rRbEM8RJEpDuvs5eWMQtblo","ver":1},"status":{"status":"write"}}},{"id":"1_1613024373.493847_99.1","section":"bucket.instance","name":"xscaju-16:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.16","timestamp":"2021-02-11T06:19:33.493847401Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rRbEM8RJEpDuvs5eWMQtblo","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024373.495976_100.1","section":"bucket","name":"xscaju-16","timestamp":"2021-02-11T06:19:33.495976000Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_sHQb-O2kysovE_jM77orOZM","ver":1},"status":{"status":"write"}}},{"id":"1_1613024373.501119_101.1","section":"bucket","name":"xscaju-16","timestamp":"2021-02-11T06:19:33.501119742Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_sHQb-O2kysovE_jM77orOZM","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024373.524093_102.1","section":"bucket.instance","name":"xscaju-17:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.17","timestamp":"2021-02-11T06:19:33.524093914Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__tpF2cPNLKiB6VVx7L7bGVh","ver":1},"status":{"status":"write"}}},{"id":"1_1613024373.529677_103.1","section":"bucket.instance","name":"xscaju-17:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.17","timestamp":"2021-02-11T06:19:33.529677757Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__tpF2cPNLKiB6VVx7L7bGVh","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024373.531628_104.1","section":"bucket","name":"xscaju-17","timestamp":"2021-02-11T06:19:33.531628576Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_EbbwHqhEqjL5axWisAmsO1p","ver":1},"status":{"status":"write"}}},{"id":"1_1613024373.538228_105.1","section":"bucket","name":"xscaju-17","timestamp":"2021-02-11T06:19:33.538228660Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_EbbwHqhEqjL5axWisAmsO1p","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024373.588815_106.1","section":"bucket.instance","name":"xscaju-18:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.18","timestamp":"2021-02-11T06:19:33.588815655Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1WW2O12ltbkT25Ghnp053dJ","ver":1},"status":{"status":"write"}}},{"id":"1_1613024373.596722_107.1","section":"bucket.instance","name":"xscaju-18:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.18","timestamp":"2021-02-11T06:19:33.596722655Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1WW2O12ltbkT25Ghnp053dJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024373.598875_108.1","section":"bucket","name":"xscaju-18","timestamp":"2021-02-11T06:19:33.598875179Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XEjzOMy0gl9N4M1Va8lXg9Y","ver":1},"status":{"status":"write"}}},{"id":"1_1613024373.606297_109.1","section":"bucket","name":"xscaju-18","timestamp":"2021-02-11T06:19:33.606297952Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XEjzOMy0gl9N4M1Va8lXg9Y","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024373.634176_110.1","section":"bucket.instance","name":"xscaju-16:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.16","timestamp":"2021-02-11T06:19:33.634176115Z","data":{"read_version":{"tag":"_rRbEM8RJEpDuvs5eWMQtblo","ver":1},"write_version":{"tag":"_rRbEM8RJEpDuvs5eWMQtblo","ver":2},"status":{"status":"write"}}},{"id":"1_1613024373.640813_111.1","section":"bucket.instance","name":"xscaju-16:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.16","timestamp":"2021-02-11T06:19:33.640813726Z","data":{"read_version":{"tag":"_rRbEM8RJEpDuvs5eWMQtblo","ver":1},"write_version":{"tag":"_rRbEM8RJEpDuvs5eWMQtblo","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024373.646623_112.1","section":"bucket.instance","name":"xscaju-17:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.17","timestamp":"2021-02-11T06:19:33.646623596Z","data":{"read_version":{"tag":"__tpF2cPNLKiB6VVx7L7bGVh","ver":1},"write_version":{"tag":"__tpF2cPNLKiB6VVx7L7bGVh","ver":2},"status":{"status":"write"}}},{"id":"1_1613024373.663550_113.1","section":"bucket.instance","name":"xscaju-17:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.17","timestamp":"2021-02-11T06:19:33.663550101Z","data":{"read_version":{"tag":"__tpF2cPNLKiB6VVx7L7bGVh","ver":1},"write_version":{"tag":"__tpF2cPNLKiB6VVx7L7bGVh","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024373.683136_114.1","section":"bucket.instance","name":"xscaju-18:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.18","timestamp":"2021-02-11T06:19:33.683136588Z","data":{"read_version":{"tag":"_1WW2O12ltbkT25Ghnp053dJ","ver":1},"write_version":{"tag":"_1WW2O12ltbkT25Ghnp053dJ","ver":2},"status":{"status":"write"}}},{"id":"1_1613024373.689054_115.1","section":"bucket.instance","name":"xscaju-18:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.18","timestamp":"2021-02-11T06:19:33.689054244Z","data":{"read_version":{"tag":"_1WW2O12ltbkT25Ghnp053dJ","ver":1},"write_version":{"tag":"_1WW2O12ltbkT25Ghnp053dJ","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024511.261879_127.1","section":"bucket.instance","name":"xscaju-19:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.19","timestamp":"2021-02-11T06:21:51.261879107Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_p_eBM2glYgyD_bvacpJjzWg","ver":1},"status":{"status":"write"}}},{"id":"1_1613024511.267174_134.1","section":"bucket.instance","name":"xscaju-19:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.19","timestamp":"2021-02-11T06:21:51.267174934Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_p_eBM2glYgyD_bvacpJjzWg","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024511.268990_135.1","section":"bucket","name":"xscaju-19","timestamp":"2021-02-11T06:21:51.268990551Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KmgBNF2u5Y56huMJaZaInaQ","ver":1},"status":{"status":"write"}}},{"id":"1_1613024511.274597_136.1","section":"bucket","name":"xscaju-19","timestamp":"2021-02-11T06:21:51.274597146Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KmgBNF2u5Y56huMJaZaInaQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024511.281125_137.1","section":"bucket.instance","name":"xscaju-19:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.19","timestamp":"2021-02-11T06:21:51.281125877Z","data":{"read_version":{"tag":"_p_eBM2glYgyD_bvacpJjzWg","ver":1},"write_version":{"tag":"_p_eBM2glYgyD_bvacpJjzWg","ver":2},"status":{"status":"write"}}},{"id":"1_1613024511.287525_138.1","section":"bucket.instance","name":"xscaju-19:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.19","timestamp":"2021-02-11T06:21:51.287525217Z","data":{"read_version":{"tag":"_p_eBM2glYgyD_bvacpJjzWg","ver":1},"write_version":{"tag":"_p_eBM2glYgyD_bvacpJjzWg","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024655.788162_148.1","section":"bucket.instance","name":"xscaju-20:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.20","timestamp":"2021-02-11T06:24:15.788162830Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-CKkQFoycEeUKncKE438im2","ver":1},"status":{"status":"write"}}},{"id":"1_1613024655.793777_154.1","section":"bucket.instance","name":"xscaju-20:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.20","timestamp":"2021-02-11T06:24:15.793777725Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-CKkQFoycEeUKncKE438im2","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024655.795702_155.1","section":"bucket","name":"xscaju-20","timestamp":"2021-02-11T06:24:15.795702477Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RLExy4tiUpASD0qI4I42NP-","ver":1},"status":{"status":"write"}}},{"id":"1_1613024655.802479_156.1","section":"bucket","name":"xscaju-20","timestamp":"2021-02-11T06:24:15.802479510Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RLExy4tiUpASD0qI4I42NP-","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024660.020192_159.1","section":"bucket.instance","name":"xscaju-20:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.20","timestamp":"2021-02-11T06:24:20.020192845Z","data":{"read_version":{"tag":"_-CKkQFoycEeUKncKE438im2","ver":1},"write_version":{"tag":"_-CKkQFoycEeUKncKE438im2","ver":2},"status":{"status":"write"}}},{"id":"1_1613024660.030516_162.1","section":"bucket.instance","name":"xscaju-20:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.20","timestamp":"2021-02-11T06:24:20.030516613Z","data":{"read_version":{"tag":"_-CKkQFoycEeUKncKE438im2","ver":1},"write_version":{"tag":"_-CKkQFoycEeUKncKE438im2","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024692.032178_166.1","section":"bucket.instance","name":"xscaju-20:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.20","timestamp":"2021-02-11T06:24:52.032178508Z","data":{"read_version":{"tag":"_-CKkQFoycEeUKncKE438im2","ver":2},"write_version":{"tag":"_-CKkQFoycEeUKncKE438im2","ver":3},"status":{"status":"write"}}},{"id":"1_1613024692.042180_169.1","section":"bucket.instance","name":"xscaju-20:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.20","timestamp":"2021-02-11T06:24:52.042180753Z","data":{"read_version":{"tag":"_-CKkQFoycEeUKncKE438im2","ver":2},"write_version":{"tag":"_-CKkQFoycEeUKncKE438im2","ver":3},"status":{"status":"complete"}}},{"id":"1_1613024712.687247_172.1","section":"bucket.instance","name":"xscaju-21:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.21","timestamp":"2021-02-11T06:25:12.687247976Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_aghkHsnfiSBGUtPpitq6QYX","ver":1},"status":{"status":"write"}}},{"id":"1_1613024712.692691_175.1","section":"bucket.instance","name":"xscaju-21:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.21","timestamp":"2021-02-11T06:25:12.692691830Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_aghkHsnfiSBGUtPpitq6QYX","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024712.694732_176.1","section":"bucket","name":"xscaju-21","timestamp":"2021-02-11T06:25:12.694732388Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dEuDzfRlfV9QmLUwaDaXb1v","ver":1},"status":{"status":"write"}}},{"id":"1_1613024712.701991_177.1","section":"bucket","name":"xscaju-21","timestamp":"2021-02-11T06:25:12.701991779Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dEuDzfRlfV9QmLUwaDaXb1v","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024712.730036_178.1","section":"bucket.instance","name":"xscaju-22:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.22","timestamp":"2021-02-11T06:25:12.730036078Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9xhfxddv4lQiZriVKjsuyUG","ver":1},"status":{"status":"write"}}},{"id":"1_1613024712.735883_179.1","section":"bucket.instance","name":"xscaju-22:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.22","timestamp":"2021-02-11T06:25:12.735883165Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9xhfxddv4lQiZriVKjsuyUG","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024712.737970_180.1","section":"bucket","name":"xscaju-22","timestamp":"2021-02-11T06:25:12.737970149Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VmQT_CTFegH5L6LSM_iEfbk","ver":1},"status":{"status":"write"}}},{"id":"1_1613024712.744137_181.1","section":"bucket","name":"xscaju-22","timestamp":"2021-02-11T06:25:12.744137116Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VmQT_CTFegH5L6LSM_iEfbk","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024712.790840_182.1","section":"bucket.instance","name":"xscaju-23:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.23","timestamp":"2021-02-11T06:25:12.790840443Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_EVe05TdaPvo0mOj_E-9Ws5m","ver":1},"status":{"status":"write"}}},{"id":"1_1613024712.796616_183.1","section":"bucket.instance","name":"xscaju-23:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.23","timestamp":"2021-02-11T06:25:12.796616431Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_EVe05TdaPvo0mOj_E-9Ws5m","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024712.798717_184.1","section":"bucket","name":"xscaju-23","timestamp":"2021-02-11T06:25:12.798717915Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rk7ucCpCL0fx3vjZKmGG63K","ver":1},"status":{"status":"write"}}},{"id":"1_1613024712.805163_185.1","section":"bucket","name":"xscaju-23","timestamp":"2021-02-11T06:25:12.805163379Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rk7ucCpCL0fx3vjZKmGG63K","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024712.844997_186.1","section":"bucket.instance","name":"xscaju-21:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.21","timestamp":"2021-02-11T06:25:12.844997230Z","data":{"read_version":{"tag":"_aghkHsnfiSBGUtPpitq6QYX","ver":1},"write_version":{"tag":"_aghkHsnfiSBGUtPpitq6QYX","ver":2},"status":{"status":"write"}}},{"id":"1_1613024712.852513_187.1","section":"bucket.instance","name":"xscaju-21:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.21","timestamp":"2021-02-11T06:25:12.852513487Z","data":{"read_version":{"tag":"_aghkHsnfiSBGUtPpitq6QYX","ver":1},"write_version":{"tag":"_aghkHsnfiSBGUtPpitq6QYX","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024712.860311_188.1","section":"bucket.instance","name":"xscaju-22:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.22","timestamp":"2021-02-11T06:25:12.860311320Z","data":{"read_version":{"tag":"_9xhfxddv4lQiZriVKjsuyUG","ver":1},"write_version":{"tag":"_9xhfxddv4lQiZriVKjsuyUG","ver":2},"status":{"status":"write"}}},{"id":"1_1613024712.875536_189.1","section":"bucket.instance","name":"xscaju-22:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.22","timestamp":"2021-02-11T06:25:12.875536257Z","data":{"read_version":{"tag":"_9xhfxddv4lQiZriVKjsuyUG","ver":1},"write_version":{"tag":"_9xhfxddv4lQiZriVKjsuyUG","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024712.887831_190.1","section":"bucket.instance","name":"xscaju-23:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.23","timestamp":"2021-02-11T06:25:12.887831436Z","data":{"read_version":{"tag":"_EVe05TdaPvo0mOj_E-9Ws5m","ver":1},"write_version":{"tag":"_EVe05TdaPvo0mOj_E-9Ws5m","ver":2},"status":{"status":"write"}}},{"id":"1_1613024712.893011_191.1","section":"bucket.instance","name":"xscaju-23:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.23","timestamp":"2021-02-11T06:25:12.893011226Z","data":{"read_version":{"tag":"_EVe05TdaPvo0mOj_E-9Ws5m","ver":1},"write_version":{"tag":"_EVe05TdaPvo0mOj_E-9Ws5m","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024730.813367_209.1","section":"bucket.instance","name":"xscaju-24:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.24","timestamp":"2021-02-11T06:25:30.813367288Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fIGyWcOWpuc4VeT86y9JIXc","ver":1},"status":{"status":"write"}}},{"id":"1_1613024730.818704_214.1","section":"bucket.instance","name":"xscaju-24:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.24","timestamp":"2021-02-11T06:25:30.818704273Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fIGyWcOWpuc4VeT86y9JIXc","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024730.820590_215.1","section":"bucket","name":"xscaju-24","timestamp":"2021-02-11T06:25:30.820590878Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CdxZUOujdzmSeUwguD2f3ih","ver":1},"status":{"status":"write"}}},{"id":"1_1613024730.828498_216.1","section":"bucket","name":"xscaju-24","timestamp":"2021-02-11T06:25:30.828498037Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CdxZUOujdzmSeUwguD2f3ih","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024730.853747_217.1","section":"bucket.instance","name":"xscaju-25:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.25","timestamp":"2021-02-11T06:25:30.853747007Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1wwb-pQJBpU2ZVKZ9xscYjF","ver":1},"status":{"status":"write"}}},{"id":"1_1613024730.860417_218.1","section":"bucket.instance","name":"xscaju-25:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.25","timestamp":"2021-02-11T06:25:30.860417124Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1wwb-pQJBpU2ZVKZ9xscYjF","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024730.862570_219.1","section":"bucket","name":"xscaju-25","timestamp":"2021-02-11T06:25:30.862570482Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RO4pHUixO0cYr5mTSfAoRR2","ver":1},"status":{"status":"write"}}},{"id":"1_1613024730.869244_220.1","section":"bucket","name":"xscaju-25","timestamp":"2021-02-11T06:25:30.869244273Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RO4pHUixO0cYr5mTSfAoRR2","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024730.918298_221.1","section":"bucket.instance","name":"xscaju-26:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.26","timestamp":"2021-02-11T06:25:30.918298568Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ztmpV4a7sa_Si3ePxy8pHiQ","ver":1},"status":{"status":"write"}}},{"id":"1_1613024730.924750_222.1","section":"bucket.instance","name":"xscaju-26:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.26","timestamp":"2021-02-11T06:25:30.924750148Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ztmpV4a7sa_Si3ePxy8pHiQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024730.927008_223.1","section":"bucket","name":"xscaju-26","timestamp":"2021-02-11T06:25:30.927008043Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5hf0n5H6P98HZy7pnWOkiJS","ver":1},"status":{"status":"write"}}},{"id":"1_1613024730.933901_224.1","section":"bucket","name":"xscaju-26","timestamp":"2021-02-11T06:25:30.933901776Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5hf0n5H6P98HZy7pnWOkiJS","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024730.959079_225.1","section":"bucket.instance","name":"xscaju-24:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.24","timestamp":"2021-02-11T06:25:30.959079377Z","data":{"read_version":{"tag":"_fIGyWcOWpuc4VeT86y9JIXc","ver":1},"write_version":{"tag":"_fIGyWcOWpuc4VeT86y9JIXc","ver":2},"status":{"status":"write"}}},{"id":"1_1613024730.965401_226.1","section":"bucket.instance","name":"xscaju-24:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.24","timestamp":"2021-02-11T06:25:30.965401947Z","data":{"read_version":{"tag":"_fIGyWcOWpuc4VeT86y9JIXc","ver":1},"write_version":{"tag":"_fIGyWcOWpuc4VeT86y9JIXc","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024730.969631_227.1","section":"bucket.instance","name":"xscaju-24:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.24","timestamp":"2021-02-11T06:25:30.969631892Z","data":{"read_version":{"tag":"_fIGyWcOWpuc4VeT86y9JIXc","ver":2},"write_version":{"tag":"_fIGyWcOWpuc4VeT86y9JIXc","ver":3},"status":{"status":"write"}}},{"id":"1_1613024730.977493_228.1","section":"bucket.instance","name":"xscaju-24:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.24","timestamp":"2021-02-11T06:25:30.977493784Z","data":{"read_version":{"tag":"_fIGyWcOWpuc4VeT86y9JIXc","ver":2},"write_version":{"tag":"_fIGyWcOWpuc4VeT86y9JIXc","ver":3},"status":{"status":"complete"}}},{"id":"1_1613024730.982502_229.1","section":"bucket.instance","name":"xscaju-25:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.25","timestamp":"2021-02-11T06:25:30.982502378Z","data":{"read_version":{"tag":"_1wwb-pQJBpU2ZVKZ9xscYjF","ver":1},"write_version":{"tag":"_1wwb-pQJBpU2ZVKZ9xscYjF","ver":2},"status":{"status":"write"}}},{"id":"1_1613024730.988537_230.1","section":"bucket.instance","name":"xscaju-25:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.25","timestamp":"2021-02-11T06:25:30.988537741Z","data":{"read_version":{"tag":"_1wwb-pQJBpU2ZVKZ9xscYjF","ver":1},"write_version":{"tag":"_1wwb-pQJBpU2ZVKZ9xscYjF","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024731.000808_231.1","section":"bucket.instance","name":"xscaju-25:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.25","timestamp":"2021-02-11T06:25:31.000808166Z","data":{"read_version":{"tag":"_1wwb-pQJBpU2ZVKZ9xscYjF","ver":2},"write_version":{"tag":"_1wwb-pQJBpU2ZVKZ9xscYjF","ver":3},"status":{"status":"write"}}},{"id":"1_1613024731.008870_232.1","section":"bucket.instance","name":"xscaju-25:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.25","timestamp":"2021-02-11T06:25:31.008870886Z","data":{"read_version":{"tag":"_1wwb-pQJBpU2ZVKZ9xscYjF","ver":2},"write_version":{"tag":"_1wwb-pQJBpU2ZVKZ9xscYjF","ver":3},"status":{"status":"complete"}}},{"id":"1_1613024731.023118_233.1","section":"bucket.instance","name":"xscaju-26:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.26","timestamp":"2021-02-11T06:25:31.023118052Z","data":{"read_version":{"tag":"_ztmpV4a7sa_Si3ePxy8pHiQ","ver":1},"write_version":{"tag":"_ztmpV4a7sa_Si3ePxy8pHiQ","ver":2},"status":{"status":"write"}}},{"id":"1_1613024731.047850_234.1","section":"bucket.instance","name":"xscaju-26:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.26","timestamp":"2021-02-11T06:25:31.047850325Z","data":{"read_version":{"tag":"_ztmpV4a7sa_Si3ePxy8pHiQ","ver":1},"write_version":{"tag":"_ztmpV4a7sa_Si3ePxy8pHiQ","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024731.065201_235.1","section":"bucket.instance","name":"xscaju-26:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.26","timestamp":"2021-02-11T06:25:31.065201447Z","data":{"read_version":{"tag":"_ztmpV4a7sa_Si3ePxy8pHiQ","ver":2},"write_version":{"tag":"_ztmpV4a7sa_Si3ePxy8pHiQ","ver":3},"status":{"status":"write"}}},{"id":"1_1613024731.075841_236.1","section":"bucket.instance","name":"xscaju-26:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.26","timestamp":"2021-02-11T06:25:31.075841634Z","data":{"read_version":{"tag":"_ztmpV4a7sa_Si3ePxy8pHiQ","ver":2},"write_version":{"tag":"_ztmpV4a7sa_Si3ePxy8pHiQ","ver":3},"status":{"status":"complete"}}},{"id":"1_1613024740.384811_241.1","section":"bucket.instance","name":"xscaju-27:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.27","timestamp":"2021-02-11T06:25:40.384811683Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TrIDSZS_H7CX19SiRjfTfaj","ver":1},"status":{"status":"write"}}},{"id":"1_1613024740.391229_248.1","section":"bucket.instance","name":"xscaju-27:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.27","timestamp":"2021-02-11T06:25:40.391229729Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TrIDSZS_H7CX19SiRjfTfaj","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024740.393342_249.1","section":"bucket","name":"xscaju-27","timestamp":"2021-02-11T06:25:40.393342210Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MTFcHaBN2BPut8Vpf8NS_O4","ver":1},"status":{"status":"write"}}},{"id":"1_1613024740.400053_250.1","section":"bucket","name":"xscaju-27","timestamp":"2021-02-11T06:25:40.400053238Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MTFcHaBN2BPut8Vpf8NS_O4","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024740.431618_251.1","section":"bucket.instance","name":"xscaju-28:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.28","timestamp":"2021-02-11T06:25:40.431618192Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_E6YSJ7lYgxp56BC7g_Mr5bI","ver":1},"status":{"status":"write"}}},{"id":"1_1613024740.439180_252.1","section":"bucket.instance","name":"xscaju-28:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.28","timestamp":"2021-02-11T06:25:40.439180036Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_E6YSJ7lYgxp56BC7g_Mr5bI","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024740.441086_253.1","section":"bucket","name":"xscaju-28","timestamp":"2021-02-11T06:25:40.441086013Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CCfBuLGioQost5PsQ4vGmNg","ver":1},"status":{"status":"write"}}},{"id":"1_1613024740.446195_254.1","section":"bucket","name":"xscaju-28","timestamp":"2021-02-11T06:25:40.446195665Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CCfBuLGioQost5PsQ4vGmNg","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024740.493564_255.1","section":"bucket.instance","name":"xscaju-29:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.29","timestamp":"2021-02-11T06:25:40.493564496Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Yn0qV-p0U41AFIUolLY2mOO","ver":1},"status":{"status":"write"}}},{"id":"1_1613024740.499171_256.1","section":"bucket.instance","name":"xscaju-29:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.29","timestamp":"2021-02-11T06:25:40.499171383Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Yn0qV-p0U41AFIUolLY2mOO","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024740.501027_257.1","section":"bucket","name":"xscaju-29","timestamp":"2021-02-11T06:25:40.501027370Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GbXKtKJm-8W6BT_rbT9R-d5","ver":1},"status":{"status":"write"}}},{"id":"1_1613024740.505530_258.1","section":"bucket","name":"xscaju-29","timestamp":"2021-02-11T06:25:40.505530321Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GbXKtKJm-8W6BT_rbT9R-d5","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024740.528269_259.1","section":"bucket.instance","name":"xscaju-27:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.27","timestamp":"2021-02-11T06:25:40.528269173Z","data":{"read_version":{"tag":"_TrIDSZS_H7CX19SiRjfTfaj","ver":1},"write_version":{"tag":"_TrIDSZS_H7CX19SiRjfTfaj","ver":2},"status":{"status":"write"}}},{"id":"1_1613024740.536371_260.1","section":"bucket.instance","name":"xscaju-27:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.27","timestamp":"2021-02-11T06:25:40.536371598Z","data":{"read_version":{"tag":"_TrIDSZS_H7CX19SiRjfTfaj","ver":1},"write_version":{"tag":"_TrIDSZS_H7CX19SiRjfTfaj","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024740.587862_261.1","section":"bucket.instance","name":"xscaju-28:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.28","timestamp":"2021-02-11T06:25:40.587862611Z","data":{"read_version":{"tag":"_E6YSJ7lYgxp56BC7g_Mr5bI","ver":1},"write_version":{"tag":"_E6YSJ7lYgxp56BC7g_Mr5bI","ver":2},"status":{"status":"write"}}},{"id":"1_1613024740.598686_262.1","section":"bucket.instance","name":"xscaju-28:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.28","timestamp":"2021-02-11T06:25:40.598686969Z","data":{"read_version":{"tag":"_E6YSJ7lYgxp56BC7g_Mr5bI","ver":1},"write_version":{"tag":"_E6YSJ7lYgxp56BC7g_Mr5bI","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024740.657924_263.1","section":"bucket.instance","name":"xscaju-29:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.29","timestamp":"2021-02-11T06:25:40.657924925Z","data":{"read_version":{"tag":"_Yn0qV-p0U41AFIUolLY2mOO","ver":1},"write_version":{"tag":"_Yn0qV-p0U41AFIUolLY2mOO","ver":2},"status":{"status":"write"}}},{"id":"1_1613024740.663844_264.1","section":"bucket.instance","name":"xscaju-29:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.29","timestamp":"2021-02-11T06:25:40.663844562Z","data":{"read_version":{"tag":"_Yn0qV-p0U41AFIUolLY2mOO","ver":1},"write_version":{"tag":"_Yn0qV-p0U41AFIUolLY2mOO","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024740.732516_265.1","section":"bucket.instance","name":"xscaju-30:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.30","timestamp":"2021-02-11T06:25:40.732516256Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MpzOxgSOKBraax9y0VvUydp","ver":1},"status":{"status":"write"}}},{"id":"1_1613024740.738207_266.1","section":"bucket.instance","name":"xscaju-30:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.30","timestamp":"2021-02-11T06:25:40.738207518Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MpzOxgSOKBraax9y0VvUydp","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024740.743036_267.1","section":"bucket","name":"xscaju-30","timestamp":"2021-02-11T06:25:40.743036899Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lLyPeKn1R0RlJIRU_JnNy5C","ver":1},"status":{"status":"write"}}},{"id":"1_1613024740.748807_268.1","section":"bucket","name":"xscaju-30","timestamp":"2021-02-11T06:25:40.748807837Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lLyPeKn1R0RlJIRU_JnNy5C","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024740.778162_269.1","section":"bucket.instance","name":"xscaju-31:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.31","timestamp":"2021-02-11T06:25:40.778162307Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_B7YFWyoUW2C3eKpdBi1oTca","ver":1},"status":{"status":"write"}}},{"id":"1_1613024740.784274_270.1","section":"bucket.instance","name":"xscaju-31:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.31","timestamp":"2021-02-11T06:25:40.784274697Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_B7YFWyoUW2C3eKpdBi1oTca","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024740.786489_271.1","section":"bucket","name":"xscaju-31","timestamp":"2021-02-11T06:25:40.786489748Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6CgMvFY80o--FV0KBLIcoKv","ver":1},"status":{"status":"write"}}},{"id":"1_1613024740.792101_272.1","section":"bucket","name":"xscaju-31","timestamp":"2021-02-11T06:25:40.792101132Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6CgMvFY80o--FV0KBLIcoKv","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024740.839308_273.1","section":"bucket.instance","name":"xscaju-32:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.32","timestamp":"2021-02-11T06:25:40.839308348Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8AE67v6AOAFpGgM2PS34g7g","ver":1},"status":{"status":"write"}}},{"id":"1_1613024740.851777_274.1","section":"bucket.instance","name":"xscaju-32:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.32","timestamp":"2021-02-11T06:25:40.851777607Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8AE67v6AOAFpGgM2PS34g7g","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024740.854155_275.1","section":"bucket","name":"xscaju-32","timestamp":"2021-02-11T06:25:40.854155749Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2Wa214TB-anBV6UIn3SPc9R","ver":1},"status":{"status":"write"}}},{"id":"1_1613024740.858532_276.1","section":"bucket","name":"xscaju-32","timestamp":"2021-02-11T06:25:40.858532323Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2Wa214TB-anBV6UIn3SPc9R","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024740.885186_277.1","section":"bucket.instance","name":"xscaju-30:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.30","timestamp":"2021-02-11T06:25:40.885186525Z","data":{"read_version":{"tag":"_MpzOxgSOKBraax9y0VvUydp","ver":1},"write_version":{"tag":"_MpzOxgSOKBraax9y0VvUydp","ver":2},"status":{"status":"write"}}},{"id":"1_1613024740.890116_278.1","section":"bucket.instance","name":"xscaju-30:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.30","timestamp":"2021-02-11T06:25:40.890116133Z","data":{"read_version":{"tag":"_MpzOxgSOKBraax9y0VvUydp","ver":1},"write_version":{"tag":"_MpzOxgSOKBraax9y0VvUydp","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024740.941521_279.1","section":"bucket.instance","name":"xscaju-31:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.31","timestamp":"2021-02-11T06:25:40.941521555Z","data":{"read_version":{"tag":"_B7YFWyoUW2C3eKpdBi1oTca","ver":1},"write_version":{"tag":"_B7YFWyoUW2C3eKpdBi1oTca","ver":2},"status":{"status":"write"}}},{"id":"1_1613024740.951603_280.1","section":"bucket.instance","name":"xscaju-31:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.31","timestamp":"2021-02-11T06:25:40.951603871Z","data":{"read_version":{"tag":"_B7YFWyoUW2C3eKpdBi1oTca","ver":1},"write_version":{"tag":"_B7YFWyoUW2C3eKpdBi1oTca","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024741.009272_281.1","section":"bucket.instance","name":"xscaju-32:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.32","timestamp":"2021-02-11T06:25:41.009272905Z","data":{"read_version":{"tag":"_8AE67v6AOAFpGgM2PS34g7g","ver":1},"write_version":{"tag":"_8AE67v6AOAFpGgM2PS34g7g","ver":2},"status":{"status":"write"}}},{"id":"1_1613024741.020533_282.1","section":"bucket.instance","name":"xscaju-32:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.32","timestamp":"2021-02-11T06:25:41.020533268Z","data":{"read_version":{"tag":"_8AE67v6AOAFpGgM2PS34g7g","ver":1},"write_version":{"tag":"_8AE67v6AOAFpGgM2PS34g7g","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024741.093044_283.1","section":"bucket.instance","name":"xscaju-33:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.33","timestamp":"2021-02-11T06:25:41.093044505Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Y1XZdFxjTuQdDNeNiv617DK","ver":1},"status":{"status":"write"}}},{"id":"1_1613024741.098818_284.1","section":"bucket.instance","name":"xscaju-33:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.33","timestamp":"2021-02-11T06:25:41.098818850Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Y1XZdFxjTuQdDNeNiv617DK","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024741.100781_285.1","section":"bucket","name":"xscaju-33","timestamp":"2021-02-11T06:25:41.100781484Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZONw9WusWv7gp_9F5F9tQmt","ver":1},"status":{"status":"write"}}},{"id":"1_1613024741.107526_286.1","section":"bucket","name":"xscaju-33","timestamp":"2021-02-11T06:25:41.107526957Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZONw9WusWv7gp_9F5F9tQmt","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024741.135594_287.1","section":"bucket.instance","name":"xscaju-34:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.34","timestamp":"2021-02-11T06:25:41.135594639Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_P2tidfUDvMkQyIFsEqdFcTf","ver":1},"status":{"status":"write"}}},{"id":"1_1613024741.141179_288.1","section":"bucket.instance","name":"xscaju-34:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.34","timestamp":"2021-02-11T06:25:41.141179587Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_P2tidfUDvMkQyIFsEqdFcTf","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024741.142984_289.1","section":"bucket","name":"xscaju-34","timestamp":"2021-02-11T06:25:41.142984206Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SMUniEEMfihR-VQY03kzJxF","ver":1},"status":{"status":"write"}}},{"id":"1_1613024741.147740_290.1","section":"bucket","name":"xscaju-34","timestamp":"2021-02-11T06:25:41.147740630Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SMUniEEMfihR-VQY03kzJxF","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024741.190155_291.1","section":"bucket.instance","name":"xscaju-35:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.35","timestamp":"2021-02-11T06:25:41.190155260Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bxF-qfdodRL7uvp-Gdc5SpJ","ver":1},"status":{"status":"write"}}},{"id":"1_1613024741.195036_292.1","section":"bucket.instance","name":"xscaju-35:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.35","timestamp":"2021-02-11T06:25:41.195036848Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bxF-qfdodRL7uvp-Gdc5SpJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024741.196874_293.1","section":"bucket","name":"xscaju-35","timestamp":"2021-02-11T06:25:41.196874908Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WKxFwlqzDjam6DeHbO6LJrK","ver":1},"status":{"status":"write"}}},{"id":"1_1613024741.202314_294.1","section":"bucket","name":"xscaju-35","timestamp":"2021-02-11T06:25:41.202314216Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WKxFwlqzDjam6DeHbO6LJrK","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024741.227566_295.1","section":"bucket.instance","name":"xscaju-33:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.33","timestamp":"2021-02-11T06:25:41.227566182Z","data":{"read_version":{"tag":"_Y1XZdFxjTuQdDNeNiv617DK","ver":1},"write_version":{"tag":"_Y1XZdFxjTuQdDNeNiv617DK","ver":2},"status":{"status":"write"}}},{"id":"1_1613024741.242076_296.1","section":"bucket.instance","name":"xscaju-33:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.33","timestamp":"2021-02-11T06:25:41.242076851Z","data":{"read_version":{"tag":"_Y1XZdFxjTuQdDNeNiv617DK","ver":1},"write_version":{"tag":"_Y1XZdFxjTuQdDNeNiv617DK","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024742.295410_297.1","section":"bucket.instance","name":"xscaju-34:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.34","timestamp":"2021-02-11T06:25:42.295410472Z","data":{"read_version":{"tag":"_P2tidfUDvMkQyIFsEqdFcTf","ver":1},"write_version":{"tag":"_P2tidfUDvMkQyIFsEqdFcTf","ver":2},"status":{"status":"write"}}},{"id":"1_1613024742.305909_298.1","section":"bucket.instance","name":"xscaju-34:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.34","timestamp":"2021-02-11T06:25:42.305909460Z","data":{"read_version":{"tag":"_P2tidfUDvMkQyIFsEqdFcTf","ver":1},"write_version":{"tag":"_P2tidfUDvMkQyIFsEqdFcTf","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024743.363794_299.1","section":"bucket.instance","name":"xscaju-35:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.35","timestamp":"2021-02-11T06:25:43.363794995Z","data":{"read_version":{"tag":"_bxF-qfdodRL7uvp-Gdc5SpJ","ver":1},"write_version":{"tag":"_bxF-qfdodRL7uvp-Gdc5SpJ","ver":2},"status":{"status":"write"}}},{"id":"1_1613024743.372995_300.1","section":"bucket.instance","name":"xscaju-35:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.35","timestamp":"2021-02-11T06:25:43.372995704Z","data":{"read_version":{"tag":"_bxF-qfdodRL7uvp-Gdc5SpJ","ver":1},"write_version":{"tag":"_bxF-qfdodRL7uvp-Gdc5SpJ","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024743.446380_301.1","section":"bucket.instance","name":"xscaju-36:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.36","timestamp":"2021-02-11T06:25:43.446380603Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SKpaH3MeA1M2VhJGApg7lav","ver":1},"status":{"status":"write"}}},{"id":"1_1613024743.453017_302.1","section":"bucket.instance","name":"xscaju-36:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.36","timestamp":"2021-02-11T06:25:43.453017160Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SKpaH3MeA1M2VhJGApg7lav","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024743.455469_303.1","section":"bucket","name":"xscaju-36","timestamp":"2021-02-11T06:25:43.455469833Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_aFSmLTfOzrHU6jetaDrOjp8","ver":1},"status":{"status":"write"}}},{"id":"1_1613024743.461930_304.1","section":"bucket","name":"xscaju-36","timestamp":"2021-02-11T06:25:43.461930643Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_aFSmLTfOzrHU6jetaDrOjp8","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024743.492333_305.1","section":"bucket.instance","name":"xscaju-37:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.37","timestamp":"2021-02-11T06:25:43.492333619Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Cjc8WiHLvzVFVpjKs0fFrVG","ver":1},"status":{"status":"write"}}},{"id":"1_1613024743.497089_306.1","section":"bucket.instance","name":"xscaju-37:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.37","timestamp":"2021-02-11T06:25:43.497089816Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Cjc8WiHLvzVFVpjKs0fFrVG","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024743.498858_307.1","section":"bucket","name":"xscaju-37","timestamp":"2021-02-11T06:25:43.498858957Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7WecVdsGP5YFX1x8ViLiq1e","ver":1},"status":{"status":"write"}}},{"id":"1_1613024743.503451_308.1","section":"bucket","name":"xscaju-37","timestamp":"2021-02-11T06:25:43.503451797Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7WecVdsGP5YFX1x8ViLiq1e","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024743.542938_309.1","section":"bucket.instance","name":"xscaju-38:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.38","timestamp":"2021-02-11T06:25:43.542938880Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-tTyqozveTUc_iwYr0G-RKw","ver":1},"status":{"status":"write"}}},{"id":"1_1613024743.549571_310.1","section":"bucket.instance","name":"xscaju-38:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.38","timestamp":"2021-02-11T06:25:43.549571216Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-tTyqozveTUc_iwYr0G-RKw","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024743.551903_311.1","section":"bucket","name":"xscaju-38","timestamp":"2021-02-11T06:25:43.551903955Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_NAJwxUoUijTAXWSISc0fRgu","ver":1},"status":{"status":"write"}}},{"id":"1_1613024743.556753_312.1","section":"bucket","name":"xscaju-38","timestamp":"2021-02-11T06:25:43.556753430Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_NAJwxUoUijTAXWSISc0fRgu","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024750.350406_317.1","section":"bucket.instance","name":"xscaju-39:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.39","timestamp":"2021-02-11T06:25:50.350406396Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WCyVAjFo5P4jDOve9UWJjyf","ver":1},"status":{"status":"write"}}},{"id":"1_1613024750.358793_318.1","section":"bucket.instance","name":"xscaju-39:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.39","timestamp":"2021-02-11T06:25:50.358793089Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WCyVAjFo5P4jDOve9UWJjyf","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024750.363526_319.1","section":"bucket","name":"xscaju-39","timestamp":"2021-02-11T06:25:50.363526805Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RQmPlZMEb_u4b1XtxQnquAQ","ver":1},"status":{"status":"write"}}},{"id":"1_1613024750.374527_320.1","section":"bucket","name":"xscaju-39","timestamp":"2021-02-11T06:25:50.374527230Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RQmPlZMEb_u4b1XtxQnquAQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024750.410352_321.1","section":"bucket.instance","name":"xscaju-40:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.40","timestamp":"2021-02-11T06:25:50.410352927Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rO2kG_GMUEGxQo2TvrXh_ea","ver":1},"status":{"status":"write"}}},{"id":"1_1613024750.416134_322.1","section":"bucket.instance","name":"xscaju-40:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.40","timestamp":"2021-02-11T06:25:50.416134960Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rO2kG_GMUEGxQo2TvrXh_ea","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024750.419388_323.1","section":"bucket","name":"xscaju-40","timestamp":"2021-02-11T06:25:50.419388083Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eTxPRJ-BIJLr1m9XX9dTVfp","ver":1},"status":{"status":"write"}}},{"id":"1_1613024750.432139_324.1","section":"bucket","name":"xscaju-40","timestamp":"2021-02-11T06:25:50.432139634Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eTxPRJ-BIJLr1m9XX9dTVfp","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024750.498625_325.1","section":"bucket.instance","name":"xscaju-41:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.41","timestamp":"2021-02-11T06:25:50.498625335Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4HV51C7OPMbwPE8MWSC6HIG","ver":1},"status":{"status":"write"}}},{"id":"1_1613024750.504135_326.1","section":"bucket.instance","name":"xscaju-41:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.41","timestamp":"2021-02-11T06:25:50.504135749Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4HV51C7OPMbwPE8MWSC6HIG","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024750.505803_327.1","section":"bucket","name":"xscaju-41","timestamp":"2021-02-11T06:25:50.505803308Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FtRMvGzj-e8dRznzOVTxQzi","ver":1},"status":{"status":"write"}}},{"id":"1_1613024750.509787_328.1","section":"bucket","name":"xscaju-41","timestamp":"2021-02-11T06:25:50.509787136Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FtRMvGzj-e8dRznzOVTxQzi","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024262.067228_26.1","section":"bucket.instance","name":"xscaju-1:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.1","timestamp":"2021-02-11T06:17:42.067228038Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RvIoHza3ZLDfg2hRKd4PKsh","ver":1},"status":{"status":"write"}}},{"id":"1_1613024262.082142_30.1","section":"bucket.instance","name":"xscaju-1:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.1","timestamp":"2021-02-11T06:17:42.082142380Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RvIoHza3ZLDfg2hRKd4PKsh","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024262.084220_31.1","section":"bucket","name":"xscaju-1","timestamp":"2021-02-11T06:17:42.084220175Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TpoivCStluhaC0Hx2vRcWbz","ver":1},"status":{"status":"write"}}},{"id":"1_1613024262.088459_32.1","section":"bucket","name":"xscaju-1","timestamp":"2021-02-11T06:17:42.088459272Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TpoivCStluhaC0Hx2vRcWbz","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024262.113934_33.1","section":"bucket.instance","name":"xscaju-2:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.2","timestamp":"2021-02-11T06:17:42.113934461Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__hd20jVKBEurKa5-YHlmHCl","ver":1},"status":{"status":"write"}}},{"id":"1_1613024262.120087_34.1","section":"bucket.instance","name":"xscaju-2:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.2","timestamp":"2021-02-11T06:17:42.120087261Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__hd20jVKBEurKa5-YHlmHCl","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024262.122293_35.1","section":"bucket","name":"xscaju-2","timestamp":"2021-02-11T06:17:42.122293620Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XnfQzd3oXvAs9-JK2tnNiPr","ver":1},"status":{"status":"write"}}},{"id":"1_1613024262.127402_36.1","section":"bucket","name":"xscaju-2","timestamp":"2021-02-11T06:17:42.127402916Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XnfQzd3oXvAs9-JK2tnNiPr","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024265.332234_37.1","section":"bucket.instance","name":"xscaju-3:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.3","timestamp":"2021-02-11T06:17:45.332234271Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AWV7DltlSln-5qmsLkFyfua","ver":1},"status":{"status":"write"}}},{"id":"1_1613024265.350497_38.1","section":"bucket.instance","name":"xscaju-3:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.3","timestamp":"2021-02-11T06:17:45.350497485Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AWV7DltlSln-5qmsLkFyfua","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024265.352443_39.1","section":"bucket","name":"xscaju-3","timestamp":"2021-02-11T06:17:45.352443766Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wQHhrUn5k7F9mLuXRB8f1Xz","ver":1},"status":{"status":"write"}}},{"id":"1_1613024265.357008_40.1","section":"bucket","name":"xscaju-3","timestamp":"2021-02-11T06:17:45.357008799Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wQHhrUn5k7F9mLuXRB8f1Xz","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024279.487119_41.1","section":"bucket.instance","name":"xscaju-4:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.4","timestamp":"2021-02-11T06:17:59.487119032Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yaLqVGz15t2aAHd-7NPFL62","ver":1},"status":{"status":"write"}}},{"id":"1_1613024279.492264_43.1","section":"bucket.instance","name":"xscaju-4:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.4","timestamp":"2021-02-11T06:17:59.492264454Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yaLqVGz15t2aAHd-7NPFL62","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024279.494230_44.1","section":"bucket","name":"xscaju-4","timestamp":"2021-02-11T06:17:59.494230677Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_339NBHqlbS70ofKCpOAhJbQ","ver":1},"status":{"status":"write"}}},{"id":"1_1613024279.500216_45.1","section":"bucket","name":"xscaju-4","timestamp":"2021-02-11T06:17:59.500216604Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_339NBHqlbS70ofKCpOAhJbQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024279.522608_46.1","section":"bucket.instance","name":"xscaju-5:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.5","timestamp":"2021-02-11T06:17:59.522608088Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vkQG_Xp99-yJp-ToKtT8kg3","ver":1},"status":{"status":"write"}}},{"id":"1_1613024279.531162_47.1","section":"bucket.instance","name":"xscaju-5:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.5","timestamp":"2021-02-11T06:17:59.531162028Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vkQG_Xp99-yJp-ToKtT8kg3","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024279.533140_48.1","section":"bucket","name":"xscaju-5","timestamp":"2021-02-11T06:17:59.533140603Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SVeKSQ9gdskN7i7U-nYLEgi","ver":1},"status":{"status":"write"}}},{"id":"1_1613024279.540314_49.1","section":"bucket","name":"xscaju-5","timestamp":"2021-02-11T06:17:59.540314431Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SVeKSQ9gdskN7i7U-nYLEgi","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024279.587364_50.1","section":"bucket.instance","name":"xscaju-6:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.6","timestamp":"2021-02-11T06:17:59.587364825Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rte0LgEFVcWvYNPAx9nNs_n","ver":1},"status":{"status":"write"}}},{"id":"1_1613024279.595650_51.1","section":"bucket.instance","name":"xscaju-6:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.6","timestamp":"2021-02-11T06:17:59.595650557Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rte0LgEFVcWvYNPAx9nNs_n","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024279.599110_52.1","section":"bucket","name":"xscaju-6","timestamp":"2021-02-11T06:17:59.599110042Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9FvsuLFeId2rWrC78ZduVOd","ver":1},"status":{"status":"write"}}},{"id":"1_1613024279.610246_53.1","section":"bucket","name":"xscaju-6","timestamp":"2021-02-11T06:17:59.610246072Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9FvsuLFeId2rWrC78ZduVOd","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024294.216703_54.1","section":"bucket.instance","name":"xscaju-7:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.7","timestamp":"2021-02-11T06:18:14.216703877Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jWdNA4N3P_5daffu-K1YRGJ","ver":1},"status":{"status":"write"}}},{"id":"1_1613024294.222678_57.1","section":"bucket.instance","name":"xscaju-7:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.7","timestamp":"2021-02-11T06:18:14.222678059Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jWdNA4N3P_5daffu-K1YRGJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024294.224967_58.1","section":"bucket","name":"xscaju-7","timestamp":"2021-02-11T06:18:14.224967555Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_E45sVYzIU0DaBEmS8x7_Rdy","ver":1},"status":{"status":"write"}}},{"id":"1_1613024294.230021_59.1","section":"bucket","name":"xscaju-7","timestamp":"2021-02-11T06:18:14.230021669Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_E45sVYzIU0DaBEmS8x7_Rdy","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024294.249488_60.1","section":"bucket.instance","name":"xscaju-8:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.8","timestamp":"2021-02-11T06:18:14.249488401Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kfebcc_EHBe3uMLbBs7dDoL","ver":1},"status":{"status":"write"}}},{"id":"1_1613024294.255631_61.1","section":"bucket.instance","name":"xscaju-8:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.8","timestamp":"2021-02-11T06:18:14.255631956Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kfebcc_EHBe3uMLbBs7dDoL","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024294.257842_62.1","section":"bucket","name":"xscaju-8","timestamp":"2021-02-11T06:18:14.257842711Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Wh2l0tE943p3oAEfB-UgE1m","ver":1},"status":{"status":"write"}}},{"id":"1_1613024294.263232_63.1","section":"bucket","name":"xscaju-8","timestamp":"2021-02-11T06:18:14.263232235Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Wh2l0tE943p3oAEfB-UgE1m","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024294.307011_64.1","section":"bucket.instance","name":"xscaju-9:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.9","timestamp":"2021-02-11T06:18:14.307011641Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5js7vqwL9Cu1tRXnOerclkl","ver":1},"status":{"status":"write"}}},{"id":"1_1613024294.313905_65.1","section":"bucket.instance","name":"xscaju-9:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.9","timestamp":"2021-02-11T06:18:14.313905047Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5js7vqwL9Cu1tRXnOerclkl","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024294.315946_66.1","section":"bucket","name":"xscaju-9","timestamp":"2021-02-11T06:18:14.315946160Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UWAgQgKN2X4AJl_kFPBitiQ","ver":1},"status":{"status":"write"}}},{"id":"1_1613024294.322067_67.1","section":"bucket","name":"xscaju-9","timestamp":"2021-02-11T06:18:14.322067563Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UWAgQgKN2X4AJl_kFPBitiQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024300.918033_68.1","section":"bucket","name":"xscaju-7","timestamp":"2021-02-11T06:18:20.918033480Z","data":{"read_version":{"tag":"_E45sVYzIU0DaBEmS8x7_Rdy","ver":1},"write_version":{"tag":"_E45sVYzIU0DaBEmS8x7_Rdy","ver":2},"status":{"status":"remove"}}},{"id":"1_1613024300.924725_70.1","section":"bucket","name":"xscaju-7","timestamp":"2021-02-11T06:18:20.924725131Z","data":{"read_version":{"tag":"_E45sVYzIU0DaBEmS8x7_Rdy","ver":1},"write_version":{"tag":"_E45sVYzIU0DaBEmS8x7_Rdy","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024300.998791_71.1","section":"bucket","name":"xscaju-8","timestamp":"2021-02-11T06:18:20.998791991Z","data":{"read_version":{"tag":"_Wh2l0tE943p3oAEfB-UgE1m","ver":1},"write_version":{"tag":"_Wh2l0tE943p3oAEfB-UgE1m","ver":2},"status":{"status":"remove"}}},{"id":"1_1613024301.006963_72.1","section":"bucket","name":"xscaju-8","timestamp":"2021-02-11T06:18:21.006963296Z","data":{"read_version":{"tag":"_Wh2l0tE943p3oAEfB-UgE1m","ver":1},"write_version":{"tag":"_Wh2l0tE943p3oAEfB-UgE1m","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024301.135430_73.1","section":"bucket","name":"xscaju-9","timestamp":"2021-02-11T06:18:21.135430186Z","data":{"read_version":{"tag":"_UWAgQgKN2X4AJl_kFPBitiQ","ver":1},"write_version":{"tag":"_UWAgQgKN2X4AJl_kFPBitiQ","ver":2},"status":{"status":"remove"}}},{"id":"1_1613024301.142235_74.1","section":"bucket","name":"xscaju-9","timestamp":"2021-02-11T06:18:21.142235547Z","data":{"read_version":{"tag":"_UWAgQgKN2X4AJl_kFPBitiQ","ver":1},"write_version":{"tag":"_UWAgQgKN2X4AJl_kFPBitiQ","ver":2},"status":{"status":"complete"}}}] 2021-02-11T06:28:37.848 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '7c4a812f-81e4-4a72-acce-0ee18589648c', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:28:37.849 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '7c4a812f-81e4-4a72-acce-0ee18589648c', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:28:37.849 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 7c4a812f-81e4-4a72-acce-0ee18589648c --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:28:37.886 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:28:37.886 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:28:38.176 INFO:teuthology.orchestra.run.smithi172.stdout:[{"id":"1_1613024135.173475_43.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2021-02-11T06:15:35.173475237Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SbNMO4uv9J_5-KcMlPHXiJf","ver":1},"status":{"status":"write"}}},{"id":"1_1613024135.181851_48.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2021-02-11T06:15:35.181851493Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SbNMO4uv9J_5-KcMlPHXiJf","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024304.838286_55.1","section":"bucket.instance","name":"xscaju-10:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.10","timestamp":"2021-02-11T06:18:24.838286662Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vUDtGJ3GBQdG6kVLldnqDId","ver":1},"status":{"status":"write"}}},{"id":"1_1613024304.844317_62.1","section":"bucket.instance","name":"xscaju-10:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.10","timestamp":"2021-02-11T06:18:24.844317783Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vUDtGJ3GBQdG6kVLldnqDId","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024304.846426_63.1","section":"bucket","name":"xscaju-10","timestamp":"2021-02-11T06:18:24.846426466Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pfKfSMc5TvrJjKRiccmEVID","ver":1},"status":{"status":"write"}}},{"id":"1_1613024304.852830_64.1","section":"bucket","name":"xscaju-10","timestamp":"2021-02-11T06:18:24.852830013Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pfKfSMc5TvrJjKRiccmEVID","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024304.876275_65.1","section":"bucket.instance","name":"xscaju-11:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.11","timestamp":"2021-02-11T06:18:24.876275792Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7O37Ypf5_pArN3rvyGMahZQ","ver":1},"status":{"status":"write"}}},{"id":"1_1613024304.882604_66.1","section":"bucket.instance","name":"xscaju-11:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.11","timestamp":"2021-02-11T06:18:24.882604435Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7O37Ypf5_pArN3rvyGMahZQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024304.884842_67.1","section":"bucket","name":"xscaju-11","timestamp":"2021-02-11T06:18:24.884842370Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hC41ht1E97pbsgCJUm_8lj_","ver":1},"status":{"status":"write"}}},{"id":"1_1613024304.890580_68.1","section":"bucket","name":"xscaju-11","timestamp":"2021-02-11T06:18:24.890580142Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hC41ht1E97pbsgCJUm_8lj_","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024304.936648_69.1","section":"bucket.instance","name":"xscaju-12:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.12","timestamp":"2021-02-11T06:18:24.936648345Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rdqYRx6VnHnoYGod-A0qeUL","ver":1},"status":{"status":"write"}}},{"id":"1_1613024304.943318_70.1","section":"bucket.instance","name":"xscaju-12:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.12","timestamp":"2021-02-11T06:18:24.943318666Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rdqYRx6VnHnoYGod-A0qeUL","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024304.946295_71.1","section":"bucket","name":"xscaju-12","timestamp":"2021-02-11T06:18:24.946295639Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YM8ZNKbVwbAQ1oc9Cz4_OM_","ver":1},"status":{"status":"write"}}},{"id":"1_1613024304.951770_72.1","section":"bucket","name":"xscaju-12","timestamp":"2021-02-11T06:18:24.951770048Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YM8ZNKbVwbAQ1oc9Cz4_OM_","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024341.501587_78.1","section":"bucket.instance","name":"xscaju-13:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.13","timestamp":"2021-02-11T06:19:01.501587544Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BCFlY5FQTgcsjiMtHT5pskt","ver":1},"status":{"status":"write"}}},{"id":"1_1613024341.507190_81.1","section":"bucket.instance","name":"xscaju-13:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.13","timestamp":"2021-02-11T06:19:01.507190496Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BCFlY5FQTgcsjiMtHT5pskt","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024341.509363_82.1","section":"bucket","name":"xscaju-13","timestamp":"2021-02-11T06:19:01.509363808Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_i2XT5Un3mY87cqRlmnmKTE1","ver":1},"status":{"status":"write"}}},{"id":"1_1613024341.515079_83.1","section":"bucket","name":"xscaju-13","timestamp":"2021-02-11T06:19:01.515079048Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_i2XT5Un3mY87cqRlmnmKTE1","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024341.540023_84.1","section":"bucket.instance","name":"xscaju-14:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.14","timestamp":"2021-02-11T06:19:01.540023533Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OG9OQa6kjk0iKLLVPQDaiG-","ver":1},"status":{"status":"write"}}},{"id":"1_1613024341.546055_85.1","section":"bucket.instance","name":"xscaju-14:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.14","timestamp":"2021-02-11T06:19:01.546055113Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OG9OQa6kjk0iKLLVPQDaiG-","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024341.548182_86.1","section":"bucket","name":"xscaju-14","timestamp":"2021-02-11T06:19:01.548182976Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LolNl85pkAeXvxWPRnXMOqS","ver":1},"status":{"status":"write"}}},{"id":"1_1613024341.553747_87.1","section":"bucket","name":"xscaju-14","timestamp":"2021-02-11T06:19:01.553747682Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LolNl85pkAeXvxWPRnXMOqS","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024341.605553_88.1","section":"bucket.instance","name":"xscaju-15:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.15","timestamp":"2021-02-11T06:19:01.605553336Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UrhgCwLCqd88P1DuEAbHlw5","ver":1},"status":{"status":"write"}}},{"id":"1_1613024341.611354_89.1","section":"bucket.instance","name":"xscaju-15:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.15","timestamp":"2021-02-11T06:19:01.611354865Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UrhgCwLCqd88P1DuEAbHlw5","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024341.613945_90.1","section":"bucket","name":"xscaju-15","timestamp":"2021-02-11T06:19:01.613945279Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FYxrvNRyhtUMgtpEdrH0XpH","ver":1},"status":{"status":"write"}}},{"id":"1_1613024341.620645_91.1","section":"bucket","name":"xscaju-15","timestamp":"2021-02-11T06:19:01.620645045Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FYxrvNRyhtUMgtpEdrH0XpH","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024373.488563_97.1","section":"bucket.instance","name":"xscaju-16:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.16","timestamp":"2021-02-11T06:19:33.488563051Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rRbEM8RJEpDuvs5eWMQtblo","ver":1},"status":{"status":"write"}}},{"id":"1_1613024373.493847_99.1","section":"bucket.instance","name":"xscaju-16:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.16","timestamp":"2021-02-11T06:19:33.493847401Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rRbEM8RJEpDuvs5eWMQtblo","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024373.495976_100.1","section":"bucket","name":"xscaju-16","timestamp":"2021-02-11T06:19:33.495976000Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_sHQb-O2kysovE_jM77orOZM","ver":1},"status":{"status":"write"}}},{"id":"1_1613024373.501119_101.1","section":"bucket","name":"xscaju-16","timestamp":"2021-02-11T06:19:33.501119742Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_sHQb-O2kysovE_jM77orOZM","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024373.524093_102.1","section":"bucket.instance","name":"xscaju-17:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.17","timestamp":"2021-02-11T06:19:33.524093914Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__tpF2cPNLKiB6VVx7L7bGVh","ver":1},"status":{"status":"write"}}},{"id":"1_1613024373.529677_103.1","section":"bucket.instance","name":"xscaju-17:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.17","timestamp":"2021-02-11T06:19:33.529677757Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__tpF2cPNLKiB6VVx7L7bGVh","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024373.531628_104.1","section":"bucket","name":"xscaju-17","timestamp":"2021-02-11T06:19:33.531628576Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_EbbwHqhEqjL5axWisAmsO1p","ver":1},"status":{"status":"write"}}},{"id":"1_1613024373.538228_105.1","section":"bucket","name":"xscaju-17","timestamp":"2021-02-11T06:19:33.538228660Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_EbbwHqhEqjL5axWisAmsO1p","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024373.588815_106.1","section":"bucket.instance","name":"xscaju-18:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.18","timestamp":"2021-02-11T06:19:33.588815655Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1WW2O12ltbkT25Ghnp053dJ","ver":1},"status":{"status":"write"}}},{"id":"1_1613024373.596722_107.1","section":"bucket.instance","name":"xscaju-18:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.18","timestamp":"2021-02-11T06:19:33.596722655Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1WW2O12ltbkT25Ghnp053dJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024373.598875_108.1","section":"bucket","name":"xscaju-18","timestamp":"2021-02-11T06:19:33.598875179Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XEjzOMy0gl9N4M1Va8lXg9Y","ver":1},"status":{"status":"write"}}},{"id":"1_1613024373.606297_109.1","section":"bucket","name":"xscaju-18","timestamp":"2021-02-11T06:19:33.606297952Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XEjzOMy0gl9N4M1Va8lXg9Y","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024373.634176_110.1","section":"bucket.instance","name":"xscaju-16:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.16","timestamp":"2021-02-11T06:19:33.634176115Z","data":{"read_version":{"tag":"_rRbEM8RJEpDuvs5eWMQtblo","ver":1},"write_version":{"tag":"_rRbEM8RJEpDuvs5eWMQtblo","ver":2},"status":{"status":"write"}}},{"id":"1_1613024373.640813_111.1","section":"bucket.instance","name":"xscaju-16:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.16","timestamp":"2021-02-11T06:19:33.640813726Z","data":{"read_version":{"tag":"_rRbEM8RJEpDuvs5eWMQtblo","ver":1},"write_version":{"tag":"_rRbEM8RJEpDuvs5eWMQtblo","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024373.646623_112.1","section":"bucket.instance","name":"xscaju-17:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.17","timestamp":"2021-02-11T06:19:33.646623596Z","data":{"read_version":{"tag":"__tpF2cPNLKiB6VVx7L7bGVh","ver":1},"write_version":{"tag":"__tpF2cPNLKiB6VVx7L7bGVh","ver":2},"status":{"status":"write"}}},{"id":"1_1613024373.663550_113.1","section":"bucket.instance","name":"xscaju-17:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.17","timestamp":"2021-02-11T06:19:33.663550101Z","data":{"read_version":{"tag":"__tpF2cPNLKiB6VVx7L7bGVh","ver":1},"write_version":{"tag":"__tpF2cPNLKiB6VVx7L7bGVh","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024373.683136_114.1","section":"bucket.instance","name":"xscaju-18:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.18","timestamp":"2021-02-11T06:19:33.683136588Z","data":{"read_version":{"tag":"_1WW2O12ltbkT25Ghnp053dJ","ver":1},"write_version":{"tag":"_1WW2O12ltbkT25Ghnp053dJ","ver":2},"status":{"status":"write"}}},{"id":"1_1613024373.689054_115.1","section":"bucket.instance","name":"xscaju-18:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.18","timestamp":"2021-02-11T06:19:33.689054244Z","data":{"read_version":{"tag":"_1WW2O12ltbkT25Ghnp053dJ","ver":1},"write_version":{"tag":"_1WW2O12ltbkT25Ghnp053dJ","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024511.261879_127.1","section":"bucket.instance","name":"xscaju-19:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.19","timestamp":"2021-02-11T06:21:51.261879107Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_p_eBM2glYgyD_bvacpJjzWg","ver":1},"status":{"status":"write"}}},{"id":"1_1613024511.267174_134.1","section":"bucket.instance","name":"xscaju-19:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.19","timestamp":"2021-02-11T06:21:51.267174934Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_p_eBM2glYgyD_bvacpJjzWg","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024511.268990_135.1","section":"bucket","name":"xscaju-19","timestamp":"2021-02-11T06:21:51.268990551Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KmgBNF2u5Y56huMJaZaInaQ","ver":1},"status":{"status":"write"}}},{"id":"1_1613024511.274597_136.1","section":"bucket","name":"xscaju-19","timestamp":"2021-02-11T06:21:51.274597146Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KmgBNF2u5Y56huMJaZaInaQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024511.281125_137.1","section":"bucket.instance","name":"xscaju-19:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.19","timestamp":"2021-02-11T06:21:51.281125877Z","data":{"read_version":{"tag":"_p_eBM2glYgyD_bvacpJjzWg","ver":1},"write_version":{"tag":"_p_eBM2glYgyD_bvacpJjzWg","ver":2},"status":{"status":"write"}}},{"id":"1_1613024511.287525_138.1","section":"bucket.instance","name":"xscaju-19:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.19","timestamp":"2021-02-11T06:21:51.287525217Z","data":{"read_version":{"tag":"_p_eBM2glYgyD_bvacpJjzWg","ver":1},"write_version":{"tag":"_p_eBM2glYgyD_bvacpJjzWg","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024655.788162_148.1","section":"bucket.instance","name":"xscaju-20:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.20","timestamp":"2021-02-11T06:24:15.788162830Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-CKkQFoycEeUKncKE438im2","ver":1},"status":{"status":"write"}}},{"id":"1_1613024655.793777_154.1","section":"bucket.instance","name":"xscaju-20:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.20","timestamp":"2021-02-11T06:24:15.793777725Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-CKkQFoycEeUKncKE438im2","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024655.795702_155.1","section":"bucket","name":"xscaju-20","timestamp":"2021-02-11T06:24:15.795702477Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RLExy4tiUpASD0qI4I42NP-","ver":1},"status":{"status":"write"}}},{"id":"1_1613024655.802479_156.1","section":"bucket","name":"xscaju-20","timestamp":"2021-02-11T06:24:15.802479510Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RLExy4tiUpASD0qI4I42NP-","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024660.020192_159.1","section":"bucket.instance","name":"xscaju-20:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.20","timestamp":"2021-02-11T06:24:20.020192845Z","data":{"read_version":{"tag":"_-CKkQFoycEeUKncKE438im2","ver":1},"write_version":{"tag":"_-CKkQFoycEeUKncKE438im2","ver":2},"status":{"status":"write"}}},{"id":"1_1613024660.030516_162.1","section":"bucket.instance","name":"xscaju-20:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.20","timestamp":"2021-02-11T06:24:20.030516613Z","data":{"read_version":{"tag":"_-CKkQFoycEeUKncKE438im2","ver":1},"write_version":{"tag":"_-CKkQFoycEeUKncKE438im2","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024692.032178_166.1","section":"bucket.instance","name":"xscaju-20:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.20","timestamp":"2021-02-11T06:24:52.032178508Z","data":{"read_version":{"tag":"_-CKkQFoycEeUKncKE438im2","ver":2},"write_version":{"tag":"_-CKkQFoycEeUKncKE438im2","ver":3},"status":{"status":"write"}}},{"id":"1_1613024692.042180_169.1","section":"bucket.instance","name":"xscaju-20:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.20","timestamp":"2021-02-11T06:24:52.042180753Z","data":{"read_version":{"tag":"_-CKkQFoycEeUKncKE438im2","ver":2},"write_version":{"tag":"_-CKkQFoycEeUKncKE438im2","ver":3},"status":{"status":"complete"}}},{"id":"1_1613024712.687247_172.1","section":"bucket.instance","name":"xscaju-21:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.21","timestamp":"2021-02-11T06:25:12.687247976Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_aghkHsnfiSBGUtPpitq6QYX","ver":1},"status":{"status":"write"}}},{"id":"1_1613024712.692691_175.1","section":"bucket.instance","name":"xscaju-21:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.21","timestamp":"2021-02-11T06:25:12.692691830Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_aghkHsnfiSBGUtPpitq6QYX","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024712.694732_176.1","section":"bucket","name":"xscaju-21","timestamp":"2021-02-11T06:25:12.694732388Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dEuDzfRlfV9QmLUwaDaXb1v","ver":1},"status":{"status":"write"}}},{"id":"1_1613024712.701991_177.1","section":"bucket","name":"xscaju-21","timestamp":"2021-02-11T06:25:12.701991779Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dEuDzfRlfV9QmLUwaDaXb1v","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024712.730036_178.1","section":"bucket.instance","name":"xscaju-22:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.22","timestamp":"2021-02-11T06:25:12.730036078Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9xhfxddv4lQiZriVKjsuyUG","ver":1},"status":{"status":"write"}}},{"id":"1_1613024712.735883_179.1","section":"bucket.instance","name":"xscaju-22:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.22","timestamp":"2021-02-11T06:25:12.735883165Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9xhfxddv4lQiZriVKjsuyUG","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024712.737970_180.1","section":"bucket","name":"xscaju-22","timestamp":"2021-02-11T06:25:12.737970149Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VmQT_CTFegH5L6LSM_iEfbk","ver":1},"status":{"status":"write"}}},{"id":"1_1613024712.744137_181.1","section":"bucket","name":"xscaju-22","timestamp":"2021-02-11T06:25:12.744137116Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VmQT_CTFegH5L6LSM_iEfbk","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024712.790840_182.1","section":"bucket.instance","name":"xscaju-23:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.23","timestamp":"2021-02-11T06:25:12.790840443Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_EVe05TdaPvo0mOj_E-9Ws5m","ver":1},"status":{"status":"write"}}},{"id":"1_1613024712.796616_183.1","section":"bucket.instance","name":"xscaju-23:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.23","timestamp":"2021-02-11T06:25:12.796616431Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_EVe05TdaPvo0mOj_E-9Ws5m","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024712.798717_184.1","section":"bucket","name":"xscaju-23","timestamp":"2021-02-11T06:25:12.798717915Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rk7ucCpCL0fx3vjZKmGG63K","ver":1},"status":{"status":"write"}}},{"id":"1_1613024712.805163_185.1","section":"bucket","name":"xscaju-23","timestamp":"2021-02-11T06:25:12.805163379Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rk7ucCpCL0fx3vjZKmGG63K","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024712.844997_186.1","section":"bucket.instance","name":"xscaju-21:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.21","timestamp":"2021-02-11T06:25:12.844997230Z","data":{"read_version":{"tag":"_aghkHsnfiSBGUtPpitq6QYX","ver":1},"write_version":{"tag":"_aghkHsnfiSBGUtPpitq6QYX","ver":2},"status":{"status":"write"}}},{"id":"1_1613024712.852513_187.1","section":"bucket.instance","name":"xscaju-21:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.21","timestamp":"2021-02-11T06:25:12.852513487Z","data":{"read_version":{"tag":"_aghkHsnfiSBGUtPpitq6QYX","ver":1},"write_version":{"tag":"_aghkHsnfiSBGUtPpitq6QYX","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024712.860311_188.1","section":"bucket.instance","name":"xscaju-22:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.22","timestamp":"2021-02-11T06:25:12.860311320Z","data":{"read_version":{"tag":"_9xhfxddv4lQiZriVKjsuyUG","ver":1},"write_version":{"tag":"_9xhfxddv4lQiZriVKjsuyUG","ver":2},"status":{"status":"write"}}},{"id":"1_1613024712.875536_189.1","section":"bucket.instance","name":"xscaju-22:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.22","timestamp":"2021-02-11T06:25:12.875536257Z","data":{"read_version":{"tag":"_9xhfxddv4lQiZriVKjsuyUG","ver":1},"write_version":{"tag":"_9xhfxddv4lQiZriVKjsuyUG","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024712.887831_190.1","section":"bucket.instance","name":"xscaju-23:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.23","timestamp":"2021-02-11T06:25:12.887831436Z","data":{"read_version":{"tag":"_EVe05TdaPvo0mOj_E-9Ws5m","ver":1},"write_version":{"tag":"_EVe05TdaPvo0mOj_E-9Ws5m","ver":2},"status":{"status":"write"}}},{"id":"1_1613024712.893011_191.1","section":"bucket.instance","name":"xscaju-23:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.23","timestamp":"2021-02-11T06:25:12.893011226Z","data":{"read_version":{"tag":"_EVe05TdaPvo0mOj_E-9Ws5m","ver":1},"write_version":{"tag":"_EVe05TdaPvo0mOj_E-9Ws5m","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024730.813367_209.1","section":"bucket.instance","name":"xscaju-24:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.24","timestamp":"2021-02-11T06:25:30.813367288Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fIGyWcOWpuc4VeT86y9JIXc","ver":1},"status":{"status":"write"}}},{"id":"1_1613024730.818704_214.1","section":"bucket.instance","name":"xscaju-24:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.24","timestamp":"2021-02-11T06:25:30.818704273Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fIGyWcOWpuc4VeT86y9JIXc","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024730.820590_215.1","section":"bucket","name":"xscaju-24","timestamp":"2021-02-11T06:25:30.820590878Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CdxZUOujdzmSeUwguD2f3ih","ver":1},"status":{"status":"write"}}},{"id":"1_1613024730.828498_216.1","section":"bucket","name":"xscaju-24","timestamp":"2021-02-11T06:25:30.828498037Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CdxZUOujdzmSeUwguD2f3ih","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024730.853747_217.1","section":"bucket.instance","name":"xscaju-25:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.25","timestamp":"2021-02-11T06:25:30.853747007Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1wwb-pQJBpU2ZVKZ9xscYjF","ver":1},"status":{"status":"write"}}},{"id":"1_1613024730.860417_218.1","section":"bucket.instance","name":"xscaju-25:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.25","timestamp":"2021-02-11T06:25:30.860417124Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1wwb-pQJBpU2ZVKZ9xscYjF","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024730.862570_219.1","section":"bucket","name":"xscaju-25","timestamp":"2021-02-11T06:25:30.862570482Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RO4pHUixO0cYr5mTSfAoRR2","ver":1},"status":{"status":"write"}}},{"id":"1_1613024730.869244_220.1","section":"bucket","name":"xscaju-25","timestamp":"2021-02-11T06:25:30.869244273Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RO4pHUixO0cYr5mTSfAoRR2","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024730.918298_221.1","section":"bucket.instance","name":"xscaju-26:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.26","timestamp":"2021-02-11T06:25:30.918298568Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ztmpV4a7sa_Si3ePxy8pHiQ","ver":1},"status":{"status":"write"}}},{"id":"1_1613024730.924750_222.1","section":"bucket.instance","name":"xscaju-26:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.26","timestamp":"2021-02-11T06:25:30.924750148Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ztmpV4a7sa_Si3ePxy8pHiQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024730.927008_223.1","section":"bucket","name":"xscaju-26","timestamp":"2021-02-11T06:25:30.927008043Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5hf0n5H6P98HZy7pnWOkiJS","ver":1},"status":{"status":"write"}}},{"id":"1_1613024730.933901_224.1","section":"bucket","name":"xscaju-26","timestamp":"2021-02-11T06:25:30.933901776Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5hf0n5H6P98HZy7pnWOkiJS","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024730.959079_225.1","section":"bucket.instance","name":"xscaju-24:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.24","timestamp":"2021-02-11T06:25:30.959079377Z","data":{"read_version":{"tag":"_fIGyWcOWpuc4VeT86y9JIXc","ver":1},"write_version":{"tag":"_fIGyWcOWpuc4VeT86y9JIXc","ver":2},"status":{"status":"write"}}},{"id":"1_1613024730.965401_226.1","section":"bucket.instance","name":"xscaju-24:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.24","timestamp":"2021-02-11T06:25:30.965401947Z","data":{"read_version":{"tag":"_fIGyWcOWpuc4VeT86y9JIXc","ver":1},"write_version":{"tag":"_fIGyWcOWpuc4VeT86y9JIXc","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024730.969631_227.1","section":"bucket.instance","name":"xscaju-24:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.24","timestamp":"2021-02-11T06:25:30.969631892Z","data":{"read_version":{"tag":"_fIGyWcOWpuc4VeT86y9JIXc","ver":2},"write_version":{"tag":"_fIGyWcOWpuc4VeT86y9JIXc","ver":3},"status":{"status":"write"}}},{"id":"1_1613024730.977493_228.1","section":"bucket.instance","name":"xscaju-24:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.24","timestamp":"2021-02-11T06:25:30.977493784Z","data":{"read_version":{"tag":"_fIGyWcOWpuc4VeT86y9JIXc","ver":2},"write_version":{"tag":"_fIGyWcOWpuc4VeT86y9JIXc","ver":3},"status":{"status":"complete"}}},{"id":"1_1613024730.982502_229.1","section":"bucket.instance","name":"xscaju-25:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.25","timestamp":"2021-02-11T06:25:30.982502378Z","data":{"read_version":{"tag":"_1wwb-pQJBpU2ZVKZ9xscYjF","ver":1},"write_version":{"tag":"_1wwb-pQJBpU2ZVKZ9xscYjF","ver":2},"status":{"status":"write"}}},{"id":"1_1613024730.988537_230.1","section":"bucket.instance","name":"xscaju-25:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.25","timestamp":"2021-02-11T06:25:30.988537741Z","data":{"read_version":{"tag":"_1wwb-pQJBpU2ZVKZ9xscYjF","ver":1},"write_version":{"tag":"_1wwb-pQJBpU2ZVKZ9xscYjF","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024731.000808_231.1","section":"bucket.instance","name":"xscaju-25:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.25","timestamp":"2021-02-11T06:25:31.000808166Z","data":{"read_version":{"tag":"_1wwb-pQJBpU2ZVKZ9xscYjF","ver":2},"write_version":{"tag":"_1wwb-pQJBpU2ZVKZ9xscYjF","ver":3},"status":{"status":"write"}}},{"id":"1_1613024731.008870_232.1","section":"bucket.instance","name":"xscaju-25:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.25","timestamp":"2021-02-11T06:25:31.008870886Z","data":{"read_version":{"tag":"_1wwb-pQJBpU2ZVKZ9xscYjF","ver":2},"write_version":{"tag":"_1wwb-pQJBpU2ZVKZ9xscYjF","ver":3},"status":{"status":"complete"}}},{"id":"1_1613024731.023118_233.1","section":"bucket.instance","name":"xscaju-26:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.26","timestamp":"2021-02-11T06:25:31.023118052Z","data":{"read_version":{"tag":"_ztmpV4a7sa_Si3ePxy8pHiQ","ver":1},"write_version":{"tag":"_ztmpV4a7sa_Si3ePxy8pHiQ","ver":2},"status":{"status":"write"}}},{"id":"1_1613024731.047850_234.1","section":"bucket.instance","name":"xscaju-26:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.26","timestamp":"2021-02-11T06:25:31.047850325Z","data":{"read_version":{"tag":"_ztmpV4a7sa_Si3ePxy8pHiQ","ver":1},"write_version":{"tag":"_ztmpV4a7sa_Si3ePxy8pHiQ","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024731.065201_235.1","section":"bucket.instance","name":"xscaju-26:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.26","timestamp":"2021-02-11T06:25:31.065201447Z","data":{"read_version":{"tag":"_ztmpV4a7sa_Si3ePxy8pHiQ","ver":2},"write_version":{"tag":"_ztmpV4a7sa_Si3ePxy8pHiQ","ver":3},"status":{"status":"write"}}},{"id":"1_1613024731.075841_236.1","section":"bucket.instance","name":"xscaju-26:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.26","timestamp":"2021-02-11T06:25:31.075841634Z","data":{"read_version":{"tag":"_ztmpV4a7sa_Si3ePxy8pHiQ","ver":2},"write_version":{"tag":"_ztmpV4a7sa_Si3ePxy8pHiQ","ver":3},"status":{"status":"complete"}}},{"id":"1_1613024740.384811_241.1","section":"bucket.instance","name":"xscaju-27:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.27","timestamp":"2021-02-11T06:25:40.384811683Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TrIDSZS_H7CX19SiRjfTfaj","ver":1},"status":{"status":"write"}}},{"id":"1_1613024740.391229_248.1","section":"bucket.instance","name":"xscaju-27:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.27","timestamp":"2021-02-11T06:25:40.391229729Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TrIDSZS_H7CX19SiRjfTfaj","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024740.393342_249.1","section":"bucket","name":"xscaju-27","timestamp":"2021-02-11T06:25:40.393342210Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MTFcHaBN2BPut8Vpf8NS_O4","ver":1},"status":{"status":"write"}}},{"id":"1_1613024740.400053_250.1","section":"bucket","name":"xscaju-27","timestamp":"2021-02-11T06:25:40.400053238Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MTFcHaBN2BPut8Vpf8NS_O4","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024740.431618_251.1","section":"bucket.instance","name":"xscaju-28:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.28","timestamp":"2021-02-11T06:25:40.431618192Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_E6YSJ7lYgxp56BC7g_Mr5bI","ver":1},"status":{"status":"write"}}},{"id":"1_1613024740.439180_252.1","section":"bucket.instance","name":"xscaju-28:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.28","timestamp":"2021-02-11T06:25:40.439180036Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_E6YSJ7lYgxp56BC7g_Mr5bI","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024740.441086_253.1","section":"bucket","name":"xscaju-28","timestamp":"2021-02-11T06:25:40.441086013Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CCfBuLGioQost5PsQ4vGmNg","ver":1},"status":{"status":"write"}}},{"id":"1_1613024740.446195_254.1","section":"bucket","name":"xscaju-28","timestamp":"2021-02-11T06:25:40.446195665Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CCfBuLGioQost5PsQ4vGmNg","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024740.493564_255.1","section":"bucket.instance","name":"xscaju-29:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.29","timestamp":"2021-02-11T06:25:40.493564496Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Yn0qV-p0U41AFIUolLY2mOO","ver":1},"status":{"status":"write"}}},{"id":"1_1613024740.499171_256.1","section":"bucket.instance","name":"xscaju-29:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.29","timestamp":"2021-02-11T06:25:40.499171383Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Yn0qV-p0U41AFIUolLY2mOO","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024740.501027_257.1","section":"bucket","name":"xscaju-29","timestamp":"2021-02-11T06:25:40.501027370Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GbXKtKJm-8W6BT_rbT9R-d5","ver":1},"status":{"status":"write"}}},{"id":"1_1613024740.505530_258.1","section":"bucket","name":"xscaju-29","timestamp":"2021-02-11T06:25:40.505530321Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GbXKtKJm-8W6BT_rbT9R-d5","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024740.528269_259.1","section":"bucket.instance","name":"xscaju-27:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.27","timestamp":"2021-02-11T06:25:40.528269173Z","data":{"read_version":{"tag":"_TrIDSZS_H7CX19SiRjfTfaj","ver":1},"write_version":{"tag":"_TrIDSZS_H7CX19SiRjfTfaj","ver":2},"status":{"status":"write"}}},{"id":"1_1613024740.536371_260.1","section":"bucket.instance","name":"xscaju-27:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.27","timestamp":"2021-02-11T06:25:40.536371598Z","data":{"read_version":{"tag":"_TrIDSZS_H7CX19SiRjfTfaj","ver":1},"write_version":{"tag":"_TrIDSZS_H7CX19SiRjfTfaj","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024740.587862_261.1","section":"bucket.instance","name":"xscaju-28:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.28","timestamp":"2021-02-11T06:25:40.587862611Z","data":{"read_version":{"tag":"_E6YSJ7lYgxp56BC7g_Mr5bI","ver":1},"write_version":{"tag":"_E6YSJ7lYgxp56BC7g_Mr5bI","ver":2},"status":{"status":"write"}}},{"id":"1_1613024740.598686_262.1","section":"bucket.instance","name":"xscaju-28:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.28","timestamp":"2021-02-11T06:25:40.598686969Z","data":{"read_version":{"tag":"_E6YSJ7lYgxp56BC7g_Mr5bI","ver":1},"write_version":{"tag":"_E6YSJ7lYgxp56BC7g_Mr5bI","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024740.657924_263.1","section":"bucket.instance","name":"xscaju-29:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.29","timestamp":"2021-02-11T06:25:40.657924925Z","data":{"read_version":{"tag":"_Yn0qV-p0U41AFIUolLY2mOO","ver":1},"write_version":{"tag":"_Yn0qV-p0U41AFIUolLY2mOO","ver":2},"status":{"status":"write"}}},{"id":"1_1613024740.663844_264.1","section":"bucket.instance","name":"xscaju-29:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.29","timestamp":"2021-02-11T06:25:40.663844562Z","data":{"read_version":{"tag":"_Yn0qV-p0U41AFIUolLY2mOO","ver":1},"write_version":{"tag":"_Yn0qV-p0U41AFIUolLY2mOO","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024740.732516_265.1","section":"bucket.instance","name":"xscaju-30:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.30","timestamp":"2021-02-11T06:25:40.732516256Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MpzOxgSOKBraax9y0VvUydp","ver":1},"status":{"status":"write"}}},{"id":"1_1613024740.738207_266.1","section":"bucket.instance","name":"xscaju-30:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.30","timestamp":"2021-02-11T06:25:40.738207518Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MpzOxgSOKBraax9y0VvUydp","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024740.743036_267.1","section":"bucket","name":"xscaju-30","timestamp":"2021-02-11T06:25:40.743036899Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lLyPeKn1R0RlJIRU_JnNy5C","ver":1},"status":{"status":"write"}}},{"id":"1_1613024740.748807_268.1","section":"bucket","name":"xscaju-30","timestamp":"2021-02-11T06:25:40.748807837Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lLyPeKn1R0RlJIRU_JnNy5C","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024740.778162_269.1","section":"bucket.instance","name":"xscaju-31:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.31","timestamp":"2021-02-11T06:25:40.778162307Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_B7YFWyoUW2C3eKpdBi1oTca","ver":1},"status":{"status":"write"}}},{"id":"1_1613024740.784274_270.1","section":"bucket.instance","name":"xscaju-31:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.31","timestamp":"2021-02-11T06:25:40.784274697Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_B7YFWyoUW2C3eKpdBi1oTca","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024740.786489_271.1","section":"bucket","name":"xscaju-31","timestamp":"2021-02-11T06:25:40.786489748Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6CgMvFY80o--FV0KBLIcoKv","ver":1},"status":{"status":"write"}}},{"id":"1_1613024740.792101_272.1","section":"bucket","name":"xscaju-31","timestamp":"2021-02-11T06:25:40.792101132Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6CgMvFY80o--FV0KBLIcoKv","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024740.839308_273.1","section":"bucket.instance","name":"xscaju-32:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.32","timestamp":"2021-02-11T06:25:40.839308348Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8AE67v6AOAFpGgM2PS34g7g","ver":1},"status":{"status":"write"}}},{"id":"1_1613024740.851777_274.1","section":"bucket.instance","name":"xscaju-32:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.32","timestamp":"2021-02-11T06:25:40.851777607Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8AE67v6AOAFpGgM2PS34g7g","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024740.854155_275.1","section":"bucket","name":"xscaju-32","timestamp":"2021-02-11T06:25:40.854155749Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2Wa214TB-anBV6UIn3SPc9R","ver":1},"status":{"status":"write"}}},{"id":"1_1613024740.858532_276.1","section":"bucket","name":"xscaju-32","timestamp":"2021-02-11T06:25:40.858532323Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2Wa214TB-anBV6UIn3SPc9R","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024740.885186_277.1","section":"bucket.instance","name":"xscaju-30:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.30","timestamp":"2021-02-11T06:25:40.885186525Z","data":{"read_version":{"tag":"_MpzOxgSOKBraax9y0VvUydp","ver":1},"write_version":{"tag":"_MpzOxgSOKBraax9y0VvUydp","ver":2},"status":{"status":"write"}}},{"id":"1_1613024740.890116_278.1","section":"bucket.instance","name":"xscaju-30:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.30","timestamp":"2021-02-11T06:25:40.890116133Z","data":{"read_version":{"tag":"_MpzOxgSOKBraax9y0VvUydp","ver":1},"write_version":{"tag":"_MpzOxgSOKBraax9y0VvUydp","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024740.941521_279.1","section":"bucket.instance","name":"xscaju-31:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.31","timestamp":"2021-02-11T06:25:40.941521555Z","data":{"read_version":{"tag":"_B7YFWyoUW2C3eKpdBi1oTca","ver":1},"write_version":{"tag":"_B7YFWyoUW2C3eKpdBi1oTca","ver":2},"status":{"status":"write"}}},{"id":"1_1613024740.951603_280.1","section":"bucket.instance","name":"xscaju-31:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.31","timestamp":"2021-02-11T06:25:40.951603871Z","data":{"read_version":{"tag":"_B7YFWyoUW2C3eKpdBi1oTca","ver":1},"write_version":{"tag":"_B7YFWyoUW2C3eKpdBi1oTca","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024741.009272_281.1","section":"bucket.instance","name":"xscaju-32:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.32","timestamp":"2021-02-11T06:25:41.009272905Z","data":{"read_version":{"tag":"_8AE67v6AOAFpGgM2PS34g7g","ver":1},"write_version":{"tag":"_8AE67v6AOAFpGgM2PS34g7g","ver":2},"status":{"status":"write"}}},{"id":"1_1613024741.020533_282.1","section":"bucket.instance","name":"xscaju-32:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.32","timestamp":"2021-02-11T06:25:41.020533268Z","data":{"read_version":{"tag":"_8AE67v6AOAFpGgM2PS34g7g","ver":1},"write_version":{"tag":"_8AE67v6AOAFpGgM2PS34g7g","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024741.093044_283.1","section":"bucket.instance","name":"xscaju-33:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.33","timestamp":"2021-02-11T06:25:41.093044505Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Y1XZdFxjTuQdDNeNiv617DK","ver":1},"status":{"status":"write"}}},{"id":"1_1613024741.098818_284.1","section":"bucket.instance","name":"xscaju-33:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.33","timestamp":"2021-02-11T06:25:41.098818850Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Y1XZdFxjTuQdDNeNiv617DK","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024741.100781_285.1","section":"bucket","name":"xscaju-33","timestamp":"2021-02-11T06:25:41.100781484Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZONw9WusWv7gp_9F5F9tQmt","ver":1},"status":{"status":"write"}}},{"id":"1_1613024741.107526_286.1","section":"bucket","name":"xscaju-33","timestamp":"2021-02-11T06:25:41.107526957Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZONw9WusWv7gp_9F5F9tQmt","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024741.135594_287.1","section":"bucket.instance","name":"xscaju-34:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.34","timestamp":"2021-02-11T06:25:41.135594639Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_P2tidfUDvMkQyIFsEqdFcTf","ver":1},"status":{"status":"write"}}},{"id":"1_1613024741.141179_288.1","section":"bucket.instance","name":"xscaju-34:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.34","timestamp":"2021-02-11T06:25:41.141179587Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_P2tidfUDvMkQyIFsEqdFcTf","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024741.142984_289.1","section":"bucket","name":"xscaju-34","timestamp":"2021-02-11T06:25:41.142984206Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SMUniEEMfihR-VQY03kzJxF","ver":1},"status":{"status":"write"}}},{"id":"1_1613024741.147740_290.1","section":"bucket","name":"xscaju-34","timestamp":"2021-02-11T06:25:41.147740630Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SMUniEEMfihR-VQY03kzJxF","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024741.190155_291.1","section":"bucket.instance","name":"xscaju-35:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.35","timestamp":"2021-02-11T06:25:41.190155260Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bxF-qfdodRL7uvp-Gdc5SpJ","ver":1},"status":{"status":"write"}}},{"id":"1_1613024741.195036_292.1","section":"bucket.instance","name":"xscaju-35:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.35","timestamp":"2021-02-11T06:25:41.195036848Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bxF-qfdodRL7uvp-Gdc5SpJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024741.196874_293.1","section":"bucket","name":"xscaju-35","timestamp":"2021-02-11T06:25:41.196874908Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WKxFwlqzDjam6DeHbO6LJrK","ver":1},"status":{"status":"write"}}},{"id":"1_1613024741.202314_294.1","section":"bucket","name":"xscaju-35","timestamp":"2021-02-11T06:25:41.202314216Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WKxFwlqzDjam6DeHbO6LJrK","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024741.227566_295.1","section":"bucket.instance","name":"xscaju-33:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.33","timestamp":"2021-02-11T06:25:41.227566182Z","data":{"read_version":{"tag":"_Y1XZdFxjTuQdDNeNiv617DK","ver":1},"write_version":{"tag":"_Y1XZdFxjTuQdDNeNiv617DK","ver":2},"status":{"status":"write"}}},{"id":"1_1613024741.242076_296.1","section":"bucket.instance","name":"xscaju-33:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.33","timestamp":"2021-02-11T06:25:41.242076851Z","data":{"read_version":{"tag":"_Y1XZdFxjTuQdDNeNiv617DK","ver":1},"write_version":{"tag":"_Y1XZdFxjTuQdDNeNiv617DK","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024742.295410_297.1","section":"bucket.instance","name":"xscaju-34:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.34","timestamp":"2021-02-11T06:25:42.295410472Z","data":{"read_version":{"tag":"_P2tidfUDvMkQyIFsEqdFcTf","ver":1},"write_version":{"tag":"_P2tidfUDvMkQyIFsEqdFcTf","ver":2},"status":{"status":"write"}}},{"id":"1_1613024742.305909_298.1","section":"bucket.instance","name":"xscaju-34:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.34","timestamp":"2021-02-11T06:25:42.305909460Z","data":{"read_version":{"tag":"_P2tidfUDvMkQyIFsEqdFcTf","ver":1},"write_version":{"tag":"_P2tidfUDvMkQyIFsEqdFcTf","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024743.363794_299.1","section":"bucket.instance","name":"xscaju-35:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.35","timestamp":"2021-02-11T06:25:43.363794995Z","data":{"read_version":{"tag":"_bxF-qfdodRL7uvp-Gdc5SpJ","ver":1},"write_version":{"tag":"_bxF-qfdodRL7uvp-Gdc5SpJ","ver":2},"status":{"status":"write"}}},{"id":"1_1613024743.372995_300.1","section":"bucket.instance","name":"xscaju-35:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.35","timestamp":"2021-02-11T06:25:43.372995704Z","data":{"read_version":{"tag":"_bxF-qfdodRL7uvp-Gdc5SpJ","ver":1},"write_version":{"tag":"_bxF-qfdodRL7uvp-Gdc5SpJ","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024743.446380_301.1","section":"bucket.instance","name":"xscaju-36:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.36","timestamp":"2021-02-11T06:25:43.446380603Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SKpaH3MeA1M2VhJGApg7lav","ver":1},"status":{"status":"write"}}},{"id":"1_1613024743.453017_302.1","section":"bucket.instance","name":"xscaju-36:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.36","timestamp":"2021-02-11T06:25:43.453017160Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SKpaH3MeA1M2VhJGApg7lav","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024743.455469_303.1","section":"bucket","name":"xscaju-36","timestamp":"2021-02-11T06:25:43.455469833Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_aFSmLTfOzrHU6jetaDrOjp8","ver":1},"status":{"status":"write"}}},{"id":"1_1613024743.461930_304.1","section":"bucket","name":"xscaju-36","timestamp":"2021-02-11T06:25:43.461930643Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_aFSmLTfOzrHU6jetaDrOjp8","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024743.492333_305.1","section":"bucket.instance","name":"xscaju-37:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.37","timestamp":"2021-02-11T06:25:43.492333619Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Cjc8WiHLvzVFVpjKs0fFrVG","ver":1},"status":{"status":"write"}}},{"id":"1_1613024743.497089_306.1","section":"bucket.instance","name":"xscaju-37:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.37","timestamp":"2021-02-11T06:25:43.497089816Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Cjc8WiHLvzVFVpjKs0fFrVG","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024743.498858_307.1","section":"bucket","name":"xscaju-37","timestamp":"2021-02-11T06:25:43.498858957Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7WecVdsGP5YFX1x8ViLiq1e","ver":1},"status":{"status":"write"}}},{"id":"1_1613024743.503451_308.1","section":"bucket","name":"xscaju-37","timestamp":"2021-02-11T06:25:43.503451797Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7WecVdsGP5YFX1x8ViLiq1e","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024743.542938_309.1","section":"bucket.instance","name":"xscaju-38:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.38","timestamp":"2021-02-11T06:25:43.542938880Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-tTyqozveTUc_iwYr0G-RKw","ver":1},"status":{"status":"write"}}},{"id":"1_1613024743.549571_310.1","section":"bucket.instance","name":"xscaju-38:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.38","timestamp":"2021-02-11T06:25:43.549571216Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-tTyqozveTUc_iwYr0G-RKw","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024743.551903_311.1","section":"bucket","name":"xscaju-38","timestamp":"2021-02-11T06:25:43.551903955Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_NAJwxUoUijTAXWSISc0fRgu","ver":1},"status":{"status":"write"}}},{"id":"1_1613024743.556753_312.1","section":"bucket","name":"xscaju-38","timestamp":"2021-02-11T06:25:43.556753430Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_NAJwxUoUijTAXWSISc0fRgu","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024750.350406_317.1","section":"bucket.instance","name":"xscaju-39:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.39","timestamp":"2021-02-11T06:25:50.350406396Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WCyVAjFo5P4jDOve9UWJjyf","ver":1},"status":{"status":"write"}}},{"id":"1_1613024750.358793_318.1","section":"bucket.instance","name":"xscaju-39:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.39","timestamp":"2021-02-11T06:25:50.358793089Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WCyVAjFo5P4jDOve9UWJjyf","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024750.363526_319.1","section":"bucket","name":"xscaju-39","timestamp":"2021-02-11T06:25:50.363526805Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RQmPlZMEb_u4b1XtxQnquAQ","ver":1},"status":{"status":"write"}}},{"id":"1_1613024750.374527_320.1","section":"bucket","name":"xscaju-39","timestamp":"2021-02-11T06:25:50.374527230Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RQmPlZMEb_u4b1XtxQnquAQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024750.410352_321.1","section":"bucket.instance","name":"xscaju-40:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.40","timestamp":"2021-02-11T06:25:50.410352927Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rO2kG_GMUEGxQo2TvrXh_ea","ver":1},"status":{"status":"write"}}},{"id":"1_1613024750.416134_322.1","section":"bucket.instance","name":"xscaju-40:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.40","timestamp":"2021-02-11T06:25:50.416134960Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rO2kG_GMUEGxQo2TvrXh_ea","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024750.419388_323.1","section":"bucket","name":"xscaju-40","timestamp":"2021-02-11T06:25:50.419388083Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eTxPRJ-BIJLr1m9XX9dTVfp","ver":1},"status":{"status":"write"}}},{"id":"1_1613024750.432139_324.1","section":"bucket","name":"xscaju-40","timestamp":"2021-02-11T06:25:50.432139634Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eTxPRJ-BIJLr1m9XX9dTVfp","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024750.498625_325.1","section":"bucket.instance","name":"xscaju-41:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.41","timestamp":"2021-02-11T06:25:50.498625335Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4HV51C7OPMbwPE8MWSC6HIG","ver":1},"status":{"status":"write"}}},{"id":"1_1613024750.504135_326.1","section":"bucket.instance","name":"xscaju-41:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.41","timestamp":"2021-02-11T06:25:50.504135749Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4HV51C7OPMbwPE8MWSC6HIG","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024750.505803_327.1","section":"bucket","name":"xscaju-41","timestamp":"2021-02-11T06:25:50.505803308Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FtRMvGzj-e8dRznzOVTxQzi","ver":1},"status":{"status":"write"}}},{"id":"1_1613024750.509787_328.1","section":"bucket","name":"xscaju-41","timestamp":"2021-02-11T06:25:50.509787136Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FtRMvGzj-e8dRznzOVTxQzi","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024262.067228_26.1","section":"bucket.instance","name":"xscaju-1:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.1","timestamp":"2021-02-11T06:17:42.067228038Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RvIoHza3ZLDfg2hRKd4PKsh","ver":1},"status":{"status":"write"}}},{"id":"1_1613024262.082142_30.1","section":"bucket.instance","name":"xscaju-1:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.1","timestamp":"2021-02-11T06:17:42.082142380Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RvIoHza3ZLDfg2hRKd4PKsh","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024262.084220_31.1","section":"bucket","name":"xscaju-1","timestamp":"2021-02-11T06:17:42.084220175Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TpoivCStluhaC0Hx2vRcWbz","ver":1},"status":{"status":"write"}}},{"id":"1_1613024262.088459_32.1","section":"bucket","name":"xscaju-1","timestamp":"2021-02-11T06:17:42.088459272Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TpoivCStluhaC0Hx2vRcWbz","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024262.113934_33.1","section":"bucket.instance","name":"xscaju-2:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.2","timestamp":"2021-02-11T06:17:42.113934461Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__hd20jVKBEurKa5-YHlmHCl","ver":1},"status":{"status":"write"}}},{"id":"1_1613024262.120087_34.1","section":"bucket.instance","name":"xscaju-2:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.2","timestamp":"2021-02-11T06:17:42.120087261Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__hd20jVKBEurKa5-YHlmHCl","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024262.122293_35.1","section":"bucket","name":"xscaju-2","timestamp":"2021-02-11T06:17:42.122293620Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XnfQzd3oXvAs9-JK2tnNiPr","ver":1},"status":{"status":"write"}}},{"id":"1_1613024262.127402_36.1","section":"bucket","name":"xscaju-2","timestamp":"2021-02-11T06:17:42.127402916Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XnfQzd3oXvAs9-JK2tnNiPr","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024265.332234_37.1","section":"bucket.instance","name":"xscaju-3:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.3","timestamp":"2021-02-11T06:17:45.332234271Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AWV7DltlSln-5qmsLkFyfua","ver":1},"status":{"status":"write"}}},{"id":"1_1613024265.350497_38.1","section":"bucket.instance","name":"xscaju-3:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.3","timestamp":"2021-02-11T06:17:45.350497485Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AWV7DltlSln-5qmsLkFyfua","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024265.352443_39.1","section":"bucket","name":"xscaju-3","timestamp":"2021-02-11T06:17:45.352443766Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wQHhrUn5k7F9mLuXRB8f1Xz","ver":1},"status":{"status":"write"}}},{"id":"1_1613024265.357008_40.1","section":"bucket","name":"xscaju-3","timestamp":"2021-02-11T06:17:45.357008799Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wQHhrUn5k7F9mLuXRB8f1Xz","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024279.487119_41.1","section":"bucket.instance","name":"xscaju-4:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.4","timestamp":"2021-02-11T06:17:59.487119032Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yaLqVGz15t2aAHd-7NPFL62","ver":1},"status":{"status":"write"}}},{"id":"1_1613024279.492264_43.1","section":"bucket.instance","name":"xscaju-4:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.4","timestamp":"2021-02-11T06:17:59.492264454Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yaLqVGz15t2aAHd-7NPFL62","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024279.494230_44.1","section":"bucket","name":"xscaju-4","timestamp":"2021-02-11T06:17:59.494230677Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_339NBHqlbS70ofKCpOAhJbQ","ver":1},"status":{"status":"write"}}},{"id":"1_1613024279.500216_45.1","section":"bucket","name":"xscaju-4","timestamp":"2021-02-11T06:17:59.500216604Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_339NBHqlbS70ofKCpOAhJbQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024279.522608_46.1","section":"bucket.instance","name":"xscaju-5:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.5","timestamp":"2021-02-11T06:17:59.522608088Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vkQG_Xp99-yJp-ToKtT8kg3","ver":1},"status":{"status":"write"}}},{"id":"1_1613024279.531162_47.1","section":"bucket.instance","name":"xscaju-5:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.5","timestamp":"2021-02-11T06:17:59.531162028Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vkQG_Xp99-yJp-ToKtT8kg3","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024279.533140_48.1","section":"bucket","name":"xscaju-5","timestamp":"2021-02-11T06:17:59.533140603Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SVeKSQ9gdskN7i7U-nYLEgi","ver":1},"status":{"status":"write"}}},{"id":"1_1613024279.540314_49.1","section":"bucket","name":"xscaju-5","timestamp":"2021-02-11T06:17:59.540314431Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SVeKSQ9gdskN7i7U-nYLEgi","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024279.587364_50.1","section":"bucket.instance","name":"xscaju-6:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.6","timestamp":"2021-02-11T06:17:59.587364825Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rte0LgEFVcWvYNPAx9nNs_n","ver":1},"status":{"status":"write"}}},{"id":"1_1613024279.595650_51.1","section":"bucket.instance","name":"xscaju-6:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.6","timestamp":"2021-02-11T06:17:59.595650557Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rte0LgEFVcWvYNPAx9nNs_n","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024279.599110_52.1","section":"bucket","name":"xscaju-6","timestamp":"2021-02-11T06:17:59.599110042Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9FvsuLFeId2rWrC78ZduVOd","ver":1},"status":{"status":"write"}}},{"id":"1_1613024279.610246_53.1","section":"bucket","name":"xscaju-6","timestamp":"2021-02-11T06:17:59.610246072Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9FvsuLFeId2rWrC78ZduVOd","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024294.216703_54.1","section":"bucket.instance","name":"xscaju-7:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.7","timestamp":"2021-02-11T06:18:14.216703877Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jWdNA4N3P_5daffu-K1YRGJ","ver":1},"status":{"status":"write"}}},{"id":"1_1613024294.222678_57.1","section":"bucket.instance","name":"xscaju-7:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.7","timestamp":"2021-02-11T06:18:14.222678059Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jWdNA4N3P_5daffu-K1YRGJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024294.224967_58.1","section":"bucket","name":"xscaju-7","timestamp":"2021-02-11T06:18:14.224967555Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_E45sVYzIU0DaBEmS8x7_Rdy","ver":1},"status":{"status":"write"}}},{"id":"1_1613024294.230021_59.1","section":"bucket","name":"xscaju-7","timestamp":"2021-02-11T06:18:14.230021669Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_E45sVYzIU0DaBEmS8x7_Rdy","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024294.249488_60.1","section":"bucket.instance","name":"xscaju-8:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.8","timestamp":"2021-02-11T06:18:14.249488401Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kfebcc_EHBe3uMLbBs7dDoL","ver":1},"status":{"status":"write"}}},{"id":"1_1613024294.255631_61.1","section":"bucket.instance","name":"xscaju-8:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.8","timestamp":"2021-02-11T06:18:14.255631956Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kfebcc_EHBe3uMLbBs7dDoL","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024294.257842_62.1","section":"bucket","name":"xscaju-8","timestamp":"2021-02-11T06:18:14.257842711Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Wh2l0tE943p3oAEfB-UgE1m","ver":1},"status":{"status":"write"}}},{"id":"1_1613024294.263232_63.1","section":"bucket","name":"xscaju-8","timestamp":"2021-02-11T06:18:14.263232235Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Wh2l0tE943p3oAEfB-UgE1m","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024294.307011_64.1","section":"bucket.instance","name":"xscaju-9:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.9","timestamp":"2021-02-11T06:18:14.307011641Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5js7vqwL9Cu1tRXnOerclkl","ver":1},"status":{"status":"write"}}},{"id":"1_1613024294.313905_65.1","section":"bucket.instance","name":"xscaju-9:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.9","timestamp":"2021-02-11T06:18:14.313905047Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5js7vqwL9Cu1tRXnOerclkl","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024294.315946_66.1","section":"bucket","name":"xscaju-9","timestamp":"2021-02-11T06:18:14.315946160Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UWAgQgKN2X4AJl_kFPBitiQ","ver":1},"status":{"status":"write"}}},{"id":"1_1613024294.322067_67.1","section":"bucket","name":"xscaju-9","timestamp":"2021-02-11T06:18:14.322067563Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UWAgQgKN2X4AJl_kFPBitiQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024300.918033_68.1","section":"bucket","name":"xscaju-7","timestamp":"2021-02-11T06:18:20.918033480Z","data":{"read_version":{"tag":"_E45sVYzIU0DaBEmS8x7_Rdy","ver":1},"write_version":{"tag":"_E45sVYzIU0DaBEmS8x7_Rdy","ver":2},"status":{"status":"remove"}}},{"id":"1_1613024300.924725_70.1","section":"bucket","name":"xscaju-7","timestamp":"2021-02-11T06:18:20.924725131Z","data":{"read_version":{"tag":"_E45sVYzIU0DaBEmS8x7_Rdy","ver":1},"write_version":{"tag":"_E45sVYzIU0DaBEmS8x7_Rdy","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024300.998791_71.1","section":"bucket","name":"xscaju-8","timestamp":"2021-02-11T06:18:20.998791991Z","data":{"read_version":{"tag":"_Wh2l0tE943p3oAEfB-UgE1m","ver":1},"write_version":{"tag":"_Wh2l0tE943p3oAEfB-UgE1m","ver":2},"status":{"status":"remove"}}},{"id":"1_1613024301.006963_72.1","section":"bucket","name":"xscaju-8","timestamp":"2021-02-11T06:18:21.006963296Z","data":{"read_version":{"tag":"_Wh2l0tE943p3oAEfB-UgE1m","ver":1},"write_version":{"tag":"_Wh2l0tE943p3oAEfB-UgE1m","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024301.135430_73.1","section":"bucket","name":"xscaju-9","timestamp":"2021-02-11T06:18:21.135430186Z","data":{"read_version":{"tag":"_UWAgQgKN2X4AJl_kFPBitiQ","ver":1},"write_version":{"tag":"_UWAgQgKN2X4AJl_kFPBitiQ","ver":2},"status":{"status":"remove"}}},{"id":"1_1613024301.142235_74.1","section":"bucket","name":"xscaju-9","timestamp":"2021-02-11T06:18:21.142235547Z","data":{"read_version":{"tag":"_UWAgQgKN2X4AJl_kFPBitiQ","ver":1},"write_version":{"tag":"_UWAgQgKN2X4AJl_kFPBitiQ","ver":2},"status":{"status":"complete"}}}] 2021-02-11T06:28:38.177 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '7c4a812f-81e4-4a72-acce-0ee18589648c', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:28:38.178 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '7c4a812f-81e4-4a72-acce-0ee18589648c', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:28:38.178 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 7c4a812f-81e4-4a72-acce-0ee18589648c --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:28:38.216 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:28:38.217 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:28:38.512 INFO:teuthology.orchestra.run.smithi061.stdout:[{"id":"1_1613024135.173475_43.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2021-02-11T06:15:35.173475237Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SbNMO4uv9J_5-KcMlPHXiJf","ver":1},"status":{"status":"write"}}},{"id":"1_1613024135.181851_48.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2021-02-11T06:15:35.181851493Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SbNMO4uv9J_5-KcMlPHXiJf","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024304.838286_55.1","section":"bucket.instance","name":"xscaju-10:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.10","timestamp":"2021-02-11T06:18:24.838286662Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vUDtGJ3GBQdG6kVLldnqDId","ver":1},"status":{"status":"write"}}},{"id":"1_1613024304.844317_62.1","section":"bucket.instance","name":"xscaju-10:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.10","timestamp":"2021-02-11T06:18:24.844317783Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vUDtGJ3GBQdG6kVLldnqDId","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024304.846426_63.1","section":"bucket","name":"xscaju-10","timestamp":"2021-02-11T06:18:24.846426466Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pfKfSMc5TvrJjKRiccmEVID","ver":1},"status":{"status":"write"}}},{"id":"1_1613024304.852830_64.1","section":"bucket","name":"xscaju-10","timestamp":"2021-02-11T06:18:24.852830013Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pfKfSMc5TvrJjKRiccmEVID","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024304.876275_65.1","section":"bucket.instance","name":"xscaju-11:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.11","timestamp":"2021-02-11T06:18:24.876275792Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7O37Ypf5_pArN3rvyGMahZQ","ver":1},"status":{"status":"write"}}},{"id":"1_1613024304.882604_66.1","section":"bucket.instance","name":"xscaju-11:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.11","timestamp":"2021-02-11T06:18:24.882604435Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7O37Ypf5_pArN3rvyGMahZQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024304.884842_67.1","section":"bucket","name":"xscaju-11","timestamp":"2021-02-11T06:18:24.884842370Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hC41ht1E97pbsgCJUm_8lj_","ver":1},"status":{"status":"write"}}},{"id":"1_1613024304.890580_68.1","section":"bucket","name":"xscaju-11","timestamp":"2021-02-11T06:18:24.890580142Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hC41ht1E97pbsgCJUm_8lj_","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024304.936648_69.1","section":"bucket.instance","name":"xscaju-12:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.12","timestamp":"2021-02-11T06:18:24.936648345Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rdqYRx6VnHnoYGod-A0qeUL","ver":1},"status":{"status":"write"}}},{"id":"1_1613024304.943318_70.1","section":"bucket.instance","name":"xscaju-12:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.12","timestamp":"2021-02-11T06:18:24.943318666Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rdqYRx6VnHnoYGod-A0qeUL","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024304.946295_71.1","section":"bucket","name":"xscaju-12","timestamp":"2021-02-11T06:18:24.946295639Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YM8ZNKbVwbAQ1oc9Cz4_OM_","ver":1},"status":{"status":"write"}}},{"id":"1_1613024304.951770_72.1","section":"bucket","name":"xscaju-12","timestamp":"2021-02-11T06:18:24.951770048Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YM8ZNKbVwbAQ1oc9Cz4_OM_","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024341.501587_78.1","section":"bucket.instance","name":"xscaju-13:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.13","timestamp":"2021-02-11T06:19:01.501587544Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BCFlY5FQTgcsjiMtHT5pskt","ver":1},"status":{"status":"write"}}},{"id":"1_1613024341.507190_81.1","section":"bucket.instance","name":"xscaju-13:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.13","timestamp":"2021-02-11T06:19:01.507190496Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BCFlY5FQTgcsjiMtHT5pskt","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024341.509363_82.1","section":"bucket","name":"xscaju-13","timestamp":"2021-02-11T06:19:01.509363808Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_i2XT5Un3mY87cqRlmnmKTE1","ver":1},"status":{"status":"write"}}},{"id":"1_1613024341.515079_83.1","section":"bucket","name":"xscaju-13","timestamp":"2021-02-11T06:19:01.515079048Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_i2XT5Un3mY87cqRlmnmKTE1","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024341.540023_84.1","section":"bucket.instance","name":"xscaju-14:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.14","timestamp":"2021-02-11T06:19:01.540023533Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OG9OQa6kjk0iKLLVPQDaiG-","ver":1},"status":{"status":"write"}}},{"id":"1_1613024341.546055_85.1","section":"bucket.instance","name":"xscaju-14:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.14","timestamp":"2021-02-11T06:19:01.546055113Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OG9OQa6kjk0iKLLVPQDaiG-","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024341.548182_86.1","section":"bucket","name":"xscaju-14","timestamp":"2021-02-11T06:19:01.548182976Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LolNl85pkAeXvxWPRnXMOqS","ver":1},"status":{"status":"write"}}},{"id":"1_1613024341.553747_87.1","section":"bucket","name":"xscaju-14","timestamp":"2021-02-11T06:19:01.553747682Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LolNl85pkAeXvxWPRnXMOqS","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024341.605553_88.1","section":"bucket.instance","name":"xscaju-15:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.15","timestamp":"2021-02-11T06:19:01.605553336Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UrhgCwLCqd88P1DuEAbHlw5","ver":1},"status":{"status":"write"}}},{"id":"1_1613024341.611354_89.1","section":"bucket.instance","name":"xscaju-15:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.15","timestamp":"2021-02-11T06:19:01.611354865Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UrhgCwLCqd88P1DuEAbHlw5","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024341.613945_90.1","section":"bucket","name":"xscaju-15","timestamp":"2021-02-11T06:19:01.613945279Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FYxrvNRyhtUMgtpEdrH0XpH","ver":1},"status":{"status":"write"}}},{"id":"1_1613024341.620645_91.1","section":"bucket","name":"xscaju-15","timestamp":"2021-02-11T06:19:01.620645045Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FYxrvNRyhtUMgtpEdrH0XpH","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024373.488563_97.1","section":"bucket.instance","name":"xscaju-16:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.16","timestamp":"2021-02-11T06:19:33.488563051Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rRbEM8RJEpDuvs5eWMQtblo","ver":1},"status":{"status":"write"}}},{"id":"1_1613024373.493847_99.1","section":"bucket.instance","name":"xscaju-16:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.16","timestamp":"2021-02-11T06:19:33.493847401Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rRbEM8RJEpDuvs5eWMQtblo","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024373.495976_100.1","section":"bucket","name":"xscaju-16","timestamp":"2021-02-11T06:19:33.495976000Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_sHQb-O2kysovE_jM77orOZM","ver":1},"status":{"status":"write"}}},{"id":"1_1613024373.501119_101.1","section":"bucket","name":"xscaju-16","timestamp":"2021-02-11T06:19:33.501119742Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_sHQb-O2kysovE_jM77orOZM","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024373.524093_102.1","section":"bucket.instance","name":"xscaju-17:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.17","timestamp":"2021-02-11T06:19:33.524093914Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__tpF2cPNLKiB6VVx7L7bGVh","ver":1},"status":{"status":"write"}}},{"id":"1_1613024373.529677_103.1","section":"bucket.instance","name":"xscaju-17:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.17","timestamp":"2021-02-11T06:19:33.529677757Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__tpF2cPNLKiB6VVx7L7bGVh","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024373.531628_104.1","section":"bucket","name":"xscaju-17","timestamp":"2021-02-11T06:19:33.531628576Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_EbbwHqhEqjL5axWisAmsO1p","ver":1},"status":{"status":"write"}}},{"id":"1_1613024373.538228_105.1","section":"bucket","name":"xscaju-17","timestamp":"2021-02-11T06:19:33.538228660Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_EbbwHqhEqjL5axWisAmsO1p","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024373.588815_106.1","section":"bucket.instance","name":"xscaju-18:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.18","timestamp":"2021-02-11T06:19:33.588815655Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1WW2O12ltbkT25Ghnp053dJ","ver":1},"status":{"status":"write"}}},{"id":"1_1613024373.596722_107.1","section":"bucket.instance","name":"xscaju-18:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.18","timestamp":"2021-02-11T06:19:33.596722655Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1WW2O12ltbkT25Ghnp053dJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024373.598875_108.1","section":"bucket","name":"xscaju-18","timestamp":"2021-02-11T06:19:33.598875179Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XEjzOMy0gl9N4M1Va8lXg9Y","ver":1},"status":{"status":"write"}}},{"id":"1_1613024373.606297_109.1","section":"bucket","name":"xscaju-18","timestamp":"2021-02-11T06:19:33.606297952Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XEjzOMy0gl9N4M1Va8lXg9Y","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024373.634176_110.1","section":"bucket.instance","name":"xscaju-16:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.16","timestamp":"2021-02-11T06:19:33.634176115Z","data":{"read_version":{"tag":"_rRbEM8RJEpDuvs5eWMQtblo","ver":1},"write_version":{"tag":"_rRbEM8RJEpDuvs5eWMQtblo","ver":2},"status":{"status":"write"}}},{"id":"1_1613024373.640813_111.1","section":"bucket.instance","name":"xscaju-16:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.16","timestamp":"2021-02-11T06:19:33.640813726Z","data":{"read_version":{"tag":"_rRbEM8RJEpDuvs5eWMQtblo","ver":1},"write_version":{"tag":"_rRbEM8RJEpDuvs5eWMQtblo","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024373.646623_112.1","section":"bucket.instance","name":"xscaju-17:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.17","timestamp":"2021-02-11T06:19:33.646623596Z","data":{"read_version":{"tag":"__tpF2cPNLKiB6VVx7L7bGVh","ver":1},"write_version":{"tag":"__tpF2cPNLKiB6VVx7L7bGVh","ver":2},"status":{"status":"write"}}},{"id":"1_1613024373.663550_113.1","section":"bucket.instance","name":"xscaju-17:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.17","timestamp":"2021-02-11T06:19:33.663550101Z","data":{"read_version":{"tag":"__tpF2cPNLKiB6VVx7L7bGVh","ver":1},"write_version":{"tag":"__tpF2cPNLKiB6VVx7L7bGVh","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024373.683136_114.1","section":"bucket.instance","name":"xscaju-18:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.18","timestamp":"2021-02-11T06:19:33.683136588Z","data":{"read_version":{"tag":"_1WW2O12ltbkT25Ghnp053dJ","ver":1},"write_version":{"tag":"_1WW2O12ltbkT25Ghnp053dJ","ver":2},"status":{"status":"write"}}},{"id":"1_1613024373.689054_115.1","section":"bucket.instance","name":"xscaju-18:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.18","timestamp":"2021-02-11T06:19:33.689054244Z","data":{"read_version":{"tag":"_1WW2O12ltbkT25Ghnp053dJ","ver":1},"write_version":{"tag":"_1WW2O12ltbkT25Ghnp053dJ","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024511.261879_127.1","section":"bucket.instance","name":"xscaju-19:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.19","timestamp":"2021-02-11T06:21:51.261879107Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_p_eBM2glYgyD_bvacpJjzWg","ver":1},"status":{"status":"write"}}},{"id":"1_1613024511.267174_134.1","section":"bucket.instance","name":"xscaju-19:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.19","timestamp":"2021-02-11T06:21:51.267174934Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_p_eBM2glYgyD_bvacpJjzWg","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024511.268990_135.1","section":"bucket","name":"xscaju-19","timestamp":"2021-02-11T06:21:51.268990551Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KmgBNF2u5Y56huMJaZaInaQ","ver":1},"status":{"status":"write"}}},{"id":"1_1613024511.274597_136.1","section":"bucket","name":"xscaju-19","timestamp":"2021-02-11T06:21:51.274597146Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KmgBNF2u5Y56huMJaZaInaQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024511.281125_137.1","section":"bucket.instance","name":"xscaju-19:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.19","timestamp":"2021-02-11T06:21:51.281125877Z","data":{"read_version":{"tag":"_p_eBM2glYgyD_bvacpJjzWg","ver":1},"write_version":{"tag":"_p_eBM2glYgyD_bvacpJjzWg","ver":2},"status":{"status":"write"}}},{"id":"1_1613024511.287525_138.1","section":"bucket.instance","name":"xscaju-19:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.19","timestamp":"2021-02-11T06:21:51.287525217Z","data":{"read_version":{"tag":"_p_eBM2glYgyD_bvacpJjzWg","ver":1},"write_version":{"tag":"_p_eBM2glYgyD_bvacpJjzWg","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024655.788162_148.1","section":"bucket.instance","name":"xscaju-20:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.20","timestamp":"2021-02-11T06:24:15.788162830Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-CKkQFoycEeUKncKE438im2","ver":1},"status":{"status":"write"}}},{"id":"1_1613024655.793777_154.1","section":"bucket.instance","name":"xscaju-20:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.20","timestamp":"2021-02-11T06:24:15.793777725Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-CKkQFoycEeUKncKE438im2","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024655.795702_155.1","section":"bucket","name":"xscaju-20","timestamp":"2021-02-11T06:24:15.795702477Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RLExy4tiUpASD0qI4I42NP-","ver":1},"status":{"status":"write"}}},{"id":"1_1613024655.802479_156.1","section":"bucket","name":"xscaju-20","timestamp":"2021-02-11T06:24:15.802479510Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RLExy4tiUpASD0qI4I42NP-","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024660.020192_159.1","section":"bucket.instance","name":"xscaju-20:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.20","timestamp":"2021-02-11T06:24:20.020192845Z","data":{"read_version":{"tag":"_-CKkQFoycEeUKncKE438im2","ver":1},"write_version":{"tag":"_-CKkQFoycEeUKncKE438im2","ver":2},"status":{"status":"write"}}},{"id":"1_1613024660.030516_162.1","section":"bucket.instance","name":"xscaju-20:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.20","timestamp":"2021-02-11T06:24:20.030516613Z","data":{"read_version":{"tag":"_-CKkQFoycEeUKncKE438im2","ver":1},"write_version":{"tag":"_-CKkQFoycEeUKncKE438im2","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024692.032178_166.1","section":"bucket.instance","name":"xscaju-20:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.20","timestamp":"2021-02-11T06:24:52.032178508Z","data":{"read_version":{"tag":"_-CKkQFoycEeUKncKE438im2","ver":2},"write_version":{"tag":"_-CKkQFoycEeUKncKE438im2","ver":3},"status":{"status":"write"}}},{"id":"1_1613024692.042180_169.1","section":"bucket.instance","name":"xscaju-20:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.20","timestamp":"2021-02-11T06:24:52.042180753Z","data":{"read_version":{"tag":"_-CKkQFoycEeUKncKE438im2","ver":2},"write_version":{"tag":"_-CKkQFoycEeUKncKE438im2","ver":3},"status":{"status":"complete"}}},{"id":"1_1613024712.687247_172.1","section":"bucket.instance","name":"xscaju-21:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.21","timestamp":"2021-02-11T06:25:12.687247976Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_aghkHsnfiSBGUtPpitq6QYX","ver":1},"status":{"status":"write"}}},{"id":"1_1613024712.692691_175.1","section":"bucket.instance","name":"xscaju-21:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.21","timestamp":"2021-02-11T06:25:12.692691830Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_aghkHsnfiSBGUtPpitq6QYX","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024712.694732_176.1","section":"bucket","name":"xscaju-21","timestamp":"2021-02-11T06:25:12.694732388Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dEuDzfRlfV9QmLUwaDaXb1v","ver":1},"status":{"status":"write"}}},{"id":"1_1613024712.701991_177.1","section":"bucket","name":"xscaju-21","timestamp":"2021-02-11T06:25:12.701991779Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dEuDzfRlfV9QmLUwaDaXb1v","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024712.730036_178.1","section":"bucket.instance","name":"xscaju-22:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.22","timestamp":"2021-02-11T06:25:12.730036078Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9xhfxddv4lQiZriVKjsuyUG","ver":1},"status":{"status":"write"}}},{"id":"1_1613024712.735883_179.1","section":"bucket.instance","name":"xscaju-22:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.22","timestamp":"2021-02-11T06:25:12.735883165Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9xhfxddv4lQiZriVKjsuyUG","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024712.737970_180.1","section":"bucket","name":"xscaju-22","timestamp":"2021-02-11T06:25:12.737970149Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VmQT_CTFegH5L6LSM_iEfbk","ver":1},"status":{"status":"write"}}},{"id":"1_1613024712.744137_181.1","section":"bucket","name":"xscaju-22","timestamp":"2021-02-11T06:25:12.744137116Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VmQT_CTFegH5L6LSM_iEfbk","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024712.790840_182.1","section":"bucket.instance","name":"xscaju-23:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.23","timestamp":"2021-02-11T06:25:12.790840443Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_EVe05TdaPvo0mOj_E-9Ws5m","ver":1},"status":{"status":"write"}}},{"id":"1_1613024712.796616_183.1","section":"bucket.instance","name":"xscaju-23:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.23","timestamp":"2021-02-11T06:25:12.796616431Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_EVe05TdaPvo0mOj_E-9Ws5m","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024712.798717_184.1","section":"bucket","name":"xscaju-23","timestamp":"2021-02-11T06:25:12.798717915Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rk7ucCpCL0fx3vjZKmGG63K","ver":1},"status":{"status":"write"}}},{"id":"1_1613024712.805163_185.1","section":"bucket","name":"xscaju-23","timestamp":"2021-02-11T06:25:12.805163379Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rk7ucCpCL0fx3vjZKmGG63K","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024712.844997_186.1","section":"bucket.instance","name":"xscaju-21:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.21","timestamp":"2021-02-11T06:25:12.844997230Z","data":{"read_version":{"tag":"_aghkHsnfiSBGUtPpitq6QYX","ver":1},"write_version":{"tag":"_aghkHsnfiSBGUtPpitq6QYX","ver":2},"status":{"status":"write"}}},{"id":"1_1613024712.852513_187.1","section":"bucket.instance","name":"xscaju-21:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.21","timestamp":"2021-02-11T06:25:12.852513487Z","data":{"read_version":{"tag":"_aghkHsnfiSBGUtPpitq6QYX","ver":1},"write_version":{"tag":"_aghkHsnfiSBGUtPpitq6QYX","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024712.860311_188.1","section":"bucket.instance","name":"xscaju-22:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.22","timestamp":"2021-02-11T06:25:12.860311320Z","data":{"read_version":{"tag":"_9xhfxddv4lQiZriVKjsuyUG","ver":1},"write_version":{"tag":"_9xhfxddv4lQiZriVKjsuyUG","ver":2},"status":{"status":"write"}}},{"id":"1_1613024712.875536_189.1","section":"bucket.instance","name":"xscaju-22:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.22","timestamp":"2021-02-11T06:25:12.875536257Z","data":{"read_version":{"tag":"_9xhfxddv4lQiZriVKjsuyUG","ver":1},"write_version":{"tag":"_9xhfxddv4lQiZriVKjsuyUG","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024712.887831_190.1","section":"bucket.instance","name":"xscaju-23:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.23","timestamp":"2021-02-11T06:25:12.887831436Z","data":{"read_version":{"tag":"_EVe05TdaPvo0mOj_E-9Ws5m","ver":1},"write_version":{"tag":"_EVe05TdaPvo0mOj_E-9Ws5m","ver":2},"status":{"status":"write"}}},{"id":"1_1613024712.893011_191.1","section":"bucket.instance","name":"xscaju-23:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.23","timestamp":"2021-02-11T06:25:12.893011226Z","data":{"read_version":{"tag":"_EVe05TdaPvo0mOj_E-9Ws5m","ver":1},"write_version":{"tag":"_EVe05TdaPvo0mOj_E-9Ws5m","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024730.813367_209.1","section":"bucket.instance","name":"xscaju-24:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.24","timestamp":"2021-02-11T06:25:30.813367288Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fIGyWcOWpuc4VeT86y9JIXc","ver":1},"status":{"status":"write"}}},{"id":"1_1613024730.818704_214.1","section":"bucket.instance","name":"xscaju-24:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.24","timestamp":"2021-02-11T06:25:30.818704273Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fIGyWcOWpuc4VeT86y9JIXc","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024730.820590_215.1","section":"bucket","name":"xscaju-24","timestamp":"2021-02-11T06:25:30.820590878Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CdxZUOujdzmSeUwguD2f3ih","ver":1},"status":{"status":"write"}}},{"id":"1_1613024730.828498_216.1","section":"bucket","name":"xscaju-24","timestamp":"2021-02-11T06:25:30.828498037Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CdxZUOujdzmSeUwguD2f3ih","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024730.853747_217.1","section":"bucket.instance","name":"xscaju-25:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.25","timestamp":"2021-02-11T06:25:30.853747007Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1wwb-pQJBpU2ZVKZ9xscYjF","ver":1},"status":{"status":"write"}}},{"id":"1_1613024730.860417_218.1","section":"bucket.instance","name":"xscaju-25:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.25","timestamp":"2021-02-11T06:25:30.860417124Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1wwb-pQJBpU2ZVKZ9xscYjF","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024730.862570_219.1","section":"bucket","name":"xscaju-25","timestamp":"2021-02-11T06:25:30.862570482Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RO4pHUixO0cYr5mTSfAoRR2","ver":1},"status":{"status":"write"}}},{"id":"1_1613024730.869244_220.1","section":"bucket","name":"xscaju-25","timestamp":"2021-02-11T06:25:30.869244273Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RO4pHUixO0cYr5mTSfAoRR2","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024730.918298_221.1","section":"bucket.instance","name":"xscaju-26:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.26","timestamp":"2021-02-11T06:25:30.918298568Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ztmpV4a7sa_Si3ePxy8pHiQ","ver":1},"status":{"status":"write"}}},{"id":"1_1613024730.924750_222.1","section":"bucket.instance","name":"xscaju-26:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.26","timestamp":"2021-02-11T06:25:30.924750148Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ztmpV4a7sa_Si3ePxy8pHiQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024730.927008_223.1","section":"bucket","name":"xscaju-26","timestamp":"2021-02-11T06:25:30.927008043Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5hf0n5H6P98HZy7pnWOkiJS","ver":1},"status":{"status":"write"}}},{"id":"1_1613024730.933901_224.1","section":"bucket","name":"xscaju-26","timestamp":"2021-02-11T06:25:30.933901776Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5hf0n5H6P98HZy7pnWOkiJS","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024730.959079_225.1","section":"bucket.instance","name":"xscaju-24:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.24","timestamp":"2021-02-11T06:25:30.959079377Z","data":{"read_version":{"tag":"_fIGyWcOWpuc4VeT86y9JIXc","ver":1},"write_version":{"tag":"_fIGyWcOWpuc4VeT86y9JIXc","ver":2},"status":{"status":"write"}}},{"id":"1_1613024730.965401_226.1","section":"bucket.instance","name":"xscaju-24:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.24","timestamp":"2021-02-11T06:25:30.965401947Z","data":{"read_version":{"tag":"_fIGyWcOWpuc4VeT86y9JIXc","ver":1},"write_version":{"tag":"_fIGyWcOWpuc4VeT86y9JIXc","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024730.969631_227.1","section":"bucket.instance","name":"xscaju-24:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.24","timestamp":"2021-02-11T06:25:30.969631892Z","data":{"read_version":{"tag":"_fIGyWcOWpuc4VeT86y9JIXc","ver":2},"write_version":{"tag":"_fIGyWcOWpuc4VeT86y9JIXc","ver":3},"status":{"status":"write"}}},{"id":"1_1613024730.977493_228.1","section":"bucket.instance","name":"xscaju-24:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.24","timestamp":"2021-02-11T06:25:30.977493784Z","data":{"read_version":{"tag":"_fIGyWcOWpuc4VeT86y9JIXc","ver":2},"write_version":{"tag":"_fIGyWcOWpuc4VeT86y9JIXc","ver":3},"status":{"status":"complete"}}},{"id":"1_1613024730.982502_229.1","section":"bucket.instance","name":"xscaju-25:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.25","timestamp":"2021-02-11T06:25:30.982502378Z","data":{"read_version":{"tag":"_1wwb-pQJBpU2ZVKZ9xscYjF","ver":1},"write_version":{"tag":"_1wwb-pQJBpU2ZVKZ9xscYjF","ver":2},"status":{"status":"write"}}},{"id":"1_1613024730.988537_230.1","section":"bucket.instance","name":"xscaju-25:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.25","timestamp":"2021-02-11T06:25:30.988537741Z","data":{"read_version":{"tag":"_1wwb-pQJBpU2ZVKZ9xscYjF","ver":1},"write_version":{"tag":"_1wwb-pQJBpU2ZVKZ9xscYjF","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024731.000808_231.1","section":"bucket.instance","name":"xscaju-25:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.25","timestamp":"2021-02-11T06:25:31.000808166Z","data":{"read_version":{"tag":"_1wwb-pQJBpU2ZVKZ9xscYjF","ver":2},"write_version":{"tag":"_1wwb-pQJBpU2ZVKZ9xscYjF","ver":3},"status":{"status":"write"}}},{"id":"1_1613024731.008870_232.1","section":"bucket.instance","name":"xscaju-25:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.25","timestamp":"2021-02-11T06:25:31.008870886Z","data":{"read_version":{"tag":"_1wwb-pQJBpU2ZVKZ9xscYjF","ver":2},"write_version":{"tag":"_1wwb-pQJBpU2ZVKZ9xscYjF","ver":3},"status":{"status":"complete"}}},{"id":"1_1613024731.023118_233.1","section":"bucket.instance","name":"xscaju-26:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.26","timestamp":"2021-02-11T06:25:31.023118052Z","data":{"read_version":{"tag":"_ztmpV4a7sa_Si3ePxy8pHiQ","ver":1},"write_version":{"tag":"_ztmpV4a7sa_Si3ePxy8pHiQ","ver":2},"status":{"status":"write"}}},{"id":"1_1613024731.047850_234.1","section":"bucket.instance","name":"xscaju-26:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.26","timestamp":"2021-02-11T06:25:31.047850325Z","data":{"read_version":{"tag":"_ztmpV4a7sa_Si3ePxy8pHiQ","ver":1},"write_version":{"tag":"_ztmpV4a7sa_Si3ePxy8pHiQ","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024731.065201_235.1","section":"bucket.instance","name":"xscaju-26:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.26","timestamp":"2021-02-11T06:25:31.065201447Z","data":{"read_version":{"tag":"_ztmpV4a7sa_Si3ePxy8pHiQ","ver":2},"write_version":{"tag":"_ztmpV4a7sa_Si3ePxy8pHiQ","ver":3},"status":{"status":"write"}}},{"id":"1_1613024731.075841_236.1","section":"bucket.instance","name":"xscaju-26:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.26","timestamp":"2021-02-11T06:25:31.075841634Z","data":{"read_version":{"tag":"_ztmpV4a7sa_Si3ePxy8pHiQ","ver":2},"write_version":{"tag":"_ztmpV4a7sa_Si3ePxy8pHiQ","ver":3},"status":{"status":"complete"}}},{"id":"1_1613024740.384811_241.1","section":"bucket.instance","name":"xscaju-27:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.27","timestamp":"2021-02-11T06:25:40.384811683Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TrIDSZS_H7CX19SiRjfTfaj","ver":1},"status":{"status":"write"}}},{"id":"1_1613024740.391229_248.1","section":"bucket.instance","name":"xscaju-27:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.27","timestamp":"2021-02-11T06:25:40.391229729Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TrIDSZS_H7CX19SiRjfTfaj","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024740.393342_249.1","section":"bucket","name":"xscaju-27","timestamp":"2021-02-11T06:25:40.393342210Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MTFcHaBN2BPut8Vpf8NS_O4","ver":1},"status":{"status":"write"}}},{"id":"1_1613024740.400053_250.1","section":"bucket","name":"xscaju-27","timestamp":"2021-02-11T06:25:40.400053238Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MTFcHaBN2BPut8Vpf8NS_O4","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024740.431618_251.1","section":"bucket.instance","name":"xscaju-28:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.28","timestamp":"2021-02-11T06:25:40.431618192Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_E6YSJ7lYgxp56BC7g_Mr5bI","ver":1},"status":{"status":"write"}}},{"id":"1_1613024740.439180_252.1","section":"bucket.instance","name":"xscaju-28:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.28","timestamp":"2021-02-11T06:25:40.439180036Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_E6YSJ7lYgxp56BC7g_Mr5bI","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024740.441086_253.1","section":"bucket","name":"xscaju-28","timestamp":"2021-02-11T06:25:40.441086013Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CCfBuLGioQost5PsQ4vGmNg","ver":1},"status":{"status":"write"}}},{"id":"1_1613024740.446195_254.1","section":"bucket","name":"xscaju-28","timestamp":"2021-02-11T06:25:40.446195665Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CCfBuLGioQost5PsQ4vGmNg","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024740.493564_255.1","section":"bucket.instance","name":"xscaju-29:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.29","timestamp":"2021-02-11T06:25:40.493564496Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Yn0qV-p0U41AFIUolLY2mOO","ver":1},"status":{"status":"write"}}},{"id":"1_1613024740.499171_256.1","section":"bucket.instance","name":"xscaju-29:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.29","timestamp":"2021-02-11T06:25:40.499171383Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Yn0qV-p0U41AFIUolLY2mOO","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024740.501027_257.1","section":"bucket","name":"xscaju-29","timestamp":"2021-02-11T06:25:40.501027370Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GbXKtKJm-8W6BT_rbT9R-d5","ver":1},"status":{"status":"write"}}},{"id":"1_1613024740.505530_258.1","section":"bucket","name":"xscaju-29","timestamp":"2021-02-11T06:25:40.505530321Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GbXKtKJm-8W6BT_rbT9R-d5","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024740.528269_259.1","section":"bucket.instance","name":"xscaju-27:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.27","timestamp":"2021-02-11T06:25:40.528269173Z","data":{"read_version":{"tag":"_TrIDSZS_H7CX19SiRjfTfaj","ver":1},"write_version":{"tag":"_TrIDSZS_H7CX19SiRjfTfaj","ver":2},"status":{"status":"write"}}},{"id":"1_1613024740.536371_260.1","section":"bucket.instance","name":"xscaju-27:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.27","timestamp":"2021-02-11T06:25:40.536371598Z","data":{"read_version":{"tag":"_TrIDSZS_H7CX19SiRjfTfaj","ver":1},"write_version":{"tag":"_TrIDSZS_H7CX19SiRjfTfaj","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024740.587862_261.1","section":"bucket.instance","name":"xscaju-28:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.28","timestamp":"2021-02-11T06:25:40.587862611Z","data":{"read_version":{"tag":"_E6YSJ7lYgxp56BC7g_Mr5bI","ver":1},"write_version":{"tag":"_E6YSJ7lYgxp56BC7g_Mr5bI","ver":2},"status":{"status":"write"}}},{"id":"1_1613024740.598686_262.1","section":"bucket.instance","name":"xscaju-28:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.28","timestamp":"2021-02-11T06:25:40.598686969Z","data":{"read_version":{"tag":"_E6YSJ7lYgxp56BC7g_Mr5bI","ver":1},"write_version":{"tag":"_E6YSJ7lYgxp56BC7g_Mr5bI","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024740.657924_263.1","section":"bucket.instance","name":"xscaju-29:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.29","timestamp":"2021-02-11T06:25:40.657924925Z","data":{"read_version":{"tag":"_Yn0qV-p0U41AFIUolLY2mOO","ver":1},"write_version":{"tag":"_Yn0qV-p0U41AFIUolLY2mOO","ver":2},"status":{"status":"write"}}},{"id":"1_1613024740.663844_264.1","section":"bucket.instance","name":"xscaju-29:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.29","timestamp":"2021-02-11T06:25:40.663844562Z","data":{"read_version":{"tag":"_Yn0qV-p0U41AFIUolLY2mOO","ver":1},"write_version":{"tag":"_Yn0qV-p0U41AFIUolLY2mOO","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024740.732516_265.1","section":"bucket.instance","name":"xscaju-30:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.30","timestamp":"2021-02-11T06:25:40.732516256Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MpzOxgSOKBraax9y0VvUydp","ver":1},"status":{"status":"write"}}},{"id":"1_1613024740.738207_266.1","section":"bucket.instance","name":"xscaju-30:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.30","timestamp":"2021-02-11T06:25:40.738207518Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MpzOxgSOKBraax9y0VvUydp","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024740.743036_267.1","section":"bucket","name":"xscaju-30","timestamp":"2021-02-11T06:25:40.743036899Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lLyPeKn1R0RlJIRU_JnNy5C","ver":1},"status":{"status":"write"}}},{"id":"1_1613024740.748807_268.1","section":"bucket","name":"xscaju-30","timestamp":"2021-02-11T06:25:40.748807837Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lLyPeKn1R0RlJIRU_JnNy5C","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024740.778162_269.1","section":"bucket.instance","name":"xscaju-31:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.31","timestamp":"2021-02-11T06:25:40.778162307Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_B7YFWyoUW2C3eKpdBi1oTca","ver":1},"status":{"status":"write"}}},{"id":"1_1613024740.784274_270.1","section":"bucket.instance","name":"xscaju-31:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.31","timestamp":"2021-02-11T06:25:40.784274697Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_B7YFWyoUW2C3eKpdBi1oTca","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024740.786489_271.1","section":"bucket","name":"xscaju-31","timestamp":"2021-02-11T06:25:40.786489748Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6CgMvFY80o--FV0KBLIcoKv","ver":1},"status":{"status":"write"}}},{"id":"1_1613024740.792101_272.1","section":"bucket","name":"xscaju-31","timestamp":"2021-02-11T06:25:40.792101132Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6CgMvFY80o--FV0KBLIcoKv","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024740.839308_273.1","section":"bucket.instance","name":"xscaju-32:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.32","timestamp":"2021-02-11T06:25:40.839308348Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8AE67v6AOAFpGgM2PS34g7g","ver":1},"status":{"status":"write"}}},{"id":"1_1613024740.851777_274.1","section":"bucket.instance","name":"xscaju-32:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.32","timestamp":"2021-02-11T06:25:40.851777607Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8AE67v6AOAFpGgM2PS34g7g","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024740.854155_275.1","section":"bucket","name":"xscaju-32","timestamp":"2021-02-11T06:25:40.854155749Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2Wa214TB-anBV6UIn3SPc9R","ver":1},"status":{"status":"write"}}},{"id":"1_1613024740.858532_276.1","section":"bucket","name":"xscaju-32","timestamp":"2021-02-11T06:25:40.858532323Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2Wa214TB-anBV6UIn3SPc9R","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024740.885186_277.1","section":"bucket.instance","name":"xscaju-30:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.30","timestamp":"2021-02-11T06:25:40.885186525Z","data":{"read_version":{"tag":"_MpzOxgSOKBraax9y0VvUydp","ver":1},"write_version":{"tag":"_MpzOxgSOKBraax9y0VvUydp","ver":2},"status":{"status":"write"}}},{"id":"1_1613024740.890116_278.1","section":"bucket.instance","name":"xscaju-30:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.30","timestamp":"2021-02-11T06:25:40.890116133Z","data":{"read_version":{"tag":"_MpzOxgSOKBraax9y0VvUydp","ver":1},"write_version":{"tag":"_MpzOxgSOKBraax9y0VvUydp","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024740.941521_279.1","section":"bucket.instance","name":"xscaju-31:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.31","timestamp":"2021-02-11T06:25:40.941521555Z","data":{"read_version":{"tag":"_B7YFWyoUW2C3eKpdBi1oTca","ver":1},"write_version":{"tag":"_B7YFWyoUW2C3eKpdBi1oTca","ver":2},"status":{"status":"write"}}},{"id":"1_1613024740.951603_280.1","section":"bucket.instance","name":"xscaju-31:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.31","timestamp":"2021-02-11T06:25:40.951603871Z","data":{"read_version":{"tag":"_B7YFWyoUW2C3eKpdBi1oTca","ver":1},"write_version":{"tag":"_B7YFWyoUW2C3eKpdBi1oTca","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024741.009272_281.1","section":"bucket.instance","name":"xscaju-32:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.32","timestamp":"2021-02-11T06:25:41.009272905Z","data":{"read_version":{"tag":"_8AE67v6AOAFpGgM2PS34g7g","ver":1},"write_version":{"tag":"_8AE67v6AOAFpGgM2PS34g7g","ver":2},"status":{"status":"write"}}},{"id":"1_1613024741.020533_282.1","section":"bucket.instance","name":"xscaju-32:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.32","timestamp":"2021-02-11T06:25:41.020533268Z","data":{"read_version":{"tag":"_8AE67v6AOAFpGgM2PS34g7g","ver":1},"write_version":{"tag":"_8AE67v6AOAFpGgM2PS34g7g","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024741.093044_283.1","section":"bucket.instance","name":"xscaju-33:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.33","timestamp":"2021-02-11T06:25:41.093044505Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Y1XZdFxjTuQdDNeNiv617DK","ver":1},"status":{"status":"write"}}},{"id":"1_1613024741.098818_284.1","section":"bucket.instance","name":"xscaju-33:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.33","timestamp":"2021-02-11T06:25:41.098818850Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Y1XZdFxjTuQdDNeNiv617DK","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024741.100781_285.1","section":"bucket","name":"xscaju-33","timestamp":"2021-02-11T06:25:41.100781484Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZONw9WusWv7gp_9F5F9tQmt","ver":1},"status":{"status":"write"}}},{"id":"1_1613024741.107526_286.1","section":"bucket","name":"xscaju-33","timestamp":"2021-02-11T06:25:41.107526957Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZONw9WusWv7gp_9F5F9tQmt","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024741.135594_287.1","section":"bucket.instance","name":"xscaju-34:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.34","timestamp":"2021-02-11T06:25:41.135594639Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_P2tidfUDvMkQyIFsEqdFcTf","ver":1},"status":{"status":"write"}}},{"id":"1_1613024741.141179_288.1","section":"bucket.instance","name":"xscaju-34:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.34","timestamp":"2021-02-11T06:25:41.141179587Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_P2tidfUDvMkQyIFsEqdFcTf","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024741.142984_289.1","section":"bucket","name":"xscaju-34","timestamp":"2021-02-11T06:25:41.142984206Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SMUniEEMfihR-VQY03kzJxF","ver":1},"status":{"status":"write"}}},{"id":"1_1613024741.147740_290.1","section":"bucket","name":"xscaju-34","timestamp":"2021-02-11T06:25:41.147740630Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SMUniEEMfihR-VQY03kzJxF","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024741.190155_291.1","section":"bucket.instance","name":"xscaju-35:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.35","timestamp":"2021-02-11T06:25:41.190155260Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bxF-qfdodRL7uvp-Gdc5SpJ","ver":1},"status":{"status":"write"}}},{"id":"1_1613024741.195036_292.1","section":"bucket.instance","name":"xscaju-35:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.35","timestamp":"2021-02-11T06:25:41.195036848Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bxF-qfdodRL7uvp-Gdc5SpJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024741.196874_293.1","section":"bucket","name":"xscaju-35","timestamp":"2021-02-11T06:25:41.196874908Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WKxFwlqzDjam6DeHbO6LJrK","ver":1},"status":{"status":"write"}}},{"id":"1_1613024741.202314_294.1","section":"bucket","name":"xscaju-35","timestamp":"2021-02-11T06:25:41.202314216Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WKxFwlqzDjam6DeHbO6LJrK","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024741.227566_295.1","section":"bucket.instance","name":"xscaju-33:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.33","timestamp":"2021-02-11T06:25:41.227566182Z","data":{"read_version":{"tag":"_Y1XZdFxjTuQdDNeNiv617DK","ver":1},"write_version":{"tag":"_Y1XZdFxjTuQdDNeNiv617DK","ver":2},"status":{"status":"write"}}},{"id":"1_1613024741.242076_296.1","section":"bucket.instance","name":"xscaju-33:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.33","timestamp":"2021-02-11T06:25:41.242076851Z","data":{"read_version":{"tag":"_Y1XZdFxjTuQdDNeNiv617DK","ver":1},"write_version":{"tag":"_Y1XZdFxjTuQdDNeNiv617DK","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024742.295410_297.1","section":"bucket.instance","name":"xscaju-34:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.34","timestamp":"2021-02-11T06:25:42.295410472Z","data":{"read_version":{"tag":"_P2tidfUDvMkQyIFsEqdFcTf","ver":1},"write_version":{"tag":"_P2tidfUDvMkQyIFsEqdFcTf","ver":2},"status":{"status":"write"}}},{"id":"1_1613024742.305909_298.1","section":"bucket.instance","name":"xscaju-34:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.34","timestamp":"2021-02-11T06:25:42.305909460Z","data":{"read_version":{"tag":"_P2tidfUDvMkQyIFsEqdFcTf","ver":1},"write_version":{"tag":"_P2tidfUDvMkQyIFsEqdFcTf","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024743.363794_299.1","section":"bucket.instance","name":"xscaju-35:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.35","timestamp":"2021-02-11T06:25:43.363794995Z","data":{"read_version":{"tag":"_bxF-qfdodRL7uvp-Gdc5SpJ","ver":1},"write_version":{"tag":"_bxF-qfdodRL7uvp-Gdc5SpJ","ver":2},"status":{"status":"write"}}},{"id":"1_1613024743.372995_300.1","section":"bucket.instance","name":"xscaju-35:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.35","timestamp":"2021-02-11T06:25:43.372995704Z","data":{"read_version":{"tag":"_bxF-qfdodRL7uvp-Gdc5SpJ","ver":1},"write_version":{"tag":"_bxF-qfdodRL7uvp-Gdc5SpJ","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024743.446380_301.1","section":"bucket.instance","name":"xscaju-36:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.36","timestamp":"2021-02-11T06:25:43.446380603Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SKpaH3MeA1M2VhJGApg7lav","ver":1},"status":{"status":"write"}}},{"id":"1_1613024743.453017_302.1","section":"bucket.instance","name":"xscaju-36:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.36","timestamp":"2021-02-11T06:25:43.453017160Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SKpaH3MeA1M2VhJGApg7lav","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024743.455469_303.1","section":"bucket","name":"xscaju-36","timestamp":"2021-02-11T06:25:43.455469833Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_aFSmLTfOzrHU6jetaDrOjp8","ver":1},"status":{"status":"write"}}},{"id":"1_1613024743.461930_304.1","section":"bucket","name":"xscaju-36","timestamp":"2021-02-11T06:25:43.461930643Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_aFSmLTfOzrHU6jetaDrOjp8","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024743.492333_305.1","section":"bucket.instance","name":"xscaju-37:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.37","timestamp":"2021-02-11T06:25:43.492333619Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Cjc8WiHLvzVFVpjKs0fFrVG","ver":1},"status":{"status":"write"}}},{"id":"1_1613024743.497089_306.1","section":"bucket.instance","name":"xscaju-37:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.37","timestamp":"2021-02-11T06:25:43.497089816Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Cjc8WiHLvzVFVpjKs0fFrVG","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024743.498858_307.1","section":"bucket","name":"xscaju-37","timestamp":"2021-02-11T06:25:43.498858957Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7WecVdsGP5YFX1x8ViLiq1e","ver":1},"status":{"status":"write"}}},{"id":"1_1613024743.503451_308.1","section":"bucket","name":"xscaju-37","timestamp":"2021-02-11T06:25:43.503451797Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7WecVdsGP5YFX1x8ViLiq1e","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024743.542938_309.1","section":"bucket.instance","name":"xscaju-38:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.38","timestamp":"2021-02-11T06:25:43.542938880Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-tTyqozveTUc_iwYr0G-RKw","ver":1},"status":{"status":"write"}}},{"id":"1_1613024743.549571_310.1","section":"bucket.instance","name":"xscaju-38:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.38","timestamp":"2021-02-11T06:25:43.549571216Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-tTyqozveTUc_iwYr0G-RKw","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024743.551903_311.1","section":"bucket","name":"xscaju-38","timestamp":"2021-02-11T06:25:43.551903955Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_NAJwxUoUijTAXWSISc0fRgu","ver":1},"status":{"status":"write"}}},{"id":"1_1613024743.556753_312.1","section":"bucket","name":"xscaju-38","timestamp":"2021-02-11T06:25:43.556753430Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_NAJwxUoUijTAXWSISc0fRgu","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024750.350406_317.1","section":"bucket.instance","name":"xscaju-39:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.39","timestamp":"2021-02-11T06:25:50.350406396Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WCyVAjFo5P4jDOve9UWJjyf","ver":1},"status":{"status":"write"}}},{"id":"1_1613024750.358793_318.1","section":"bucket.instance","name":"xscaju-39:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.39","timestamp":"2021-02-11T06:25:50.358793089Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WCyVAjFo5P4jDOve9UWJjyf","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024750.363526_319.1","section":"bucket","name":"xscaju-39","timestamp":"2021-02-11T06:25:50.363526805Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RQmPlZMEb_u4b1XtxQnquAQ","ver":1},"status":{"status":"write"}}},{"id":"1_1613024750.374527_320.1","section":"bucket","name":"xscaju-39","timestamp":"2021-02-11T06:25:50.374527230Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RQmPlZMEb_u4b1XtxQnquAQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024750.410352_321.1","section":"bucket.instance","name":"xscaju-40:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.40","timestamp":"2021-02-11T06:25:50.410352927Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rO2kG_GMUEGxQo2TvrXh_ea","ver":1},"status":{"status":"write"}}},{"id":"1_1613024750.416134_322.1","section":"bucket.instance","name":"xscaju-40:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.40","timestamp":"2021-02-11T06:25:50.416134960Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rO2kG_GMUEGxQo2TvrXh_ea","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024750.419388_323.1","section":"bucket","name":"xscaju-40","timestamp":"2021-02-11T06:25:50.419388083Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eTxPRJ-BIJLr1m9XX9dTVfp","ver":1},"status":{"status":"write"}}},{"id":"1_1613024750.432139_324.1","section":"bucket","name":"xscaju-40","timestamp":"2021-02-11T06:25:50.432139634Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eTxPRJ-BIJLr1m9XX9dTVfp","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024750.498625_325.1","section":"bucket.instance","name":"xscaju-41:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.41","timestamp":"2021-02-11T06:25:50.498625335Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4HV51C7OPMbwPE8MWSC6HIG","ver":1},"status":{"status":"write"}}},{"id":"1_1613024750.504135_326.1","section":"bucket.instance","name":"xscaju-41:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.41","timestamp":"2021-02-11T06:25:50.504135749Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4HV51C7OPMbwPE8MWSC6HIG","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024750.505803_327.1","section":"bucket","name":"xscaju-41","timestamp":"2021-02-11T06:25:50.505803308Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FtRMvGzj-e8dRznzOVTxQzi","ver":1},"status":{"status":"write"}}},{"id":"1_1613024750.509787_328.1","section":"bucket","name":"xscaju-41","timestamp":"2021-02-11T06:25:50.509787136Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FtRMvGzj-e8dRznzOVTxQzi","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024262.067228_26.1","section":"bucket.instance","name":"xscaju-1:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.1","timestamp":"2021-02-11T06:17:42.067228038Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RvIoHza3ZLDfg2hRKd4PKsh","ver":1},"status":{"status":"write"}}},{"id":"1_1613024262.082142_30.1","section":"bucket.instance","name":"xscaju-1:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.1","timestamp":"2021-02-11T06:17:42.082142380Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RvIoHza3ZLDfg2hRKd4PKsh","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024262.084220_31.1","section":"bucket","name":"xscaju-1","timestamp":"2021-02-11T06:17:42.084220175Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TpoivCStluhaC0Hx2vRcWbz","ver":1},"status":{"status":"write"}}},{"id":"1_1613024262.088459_32.1","section":"bucket","name":"xscaju-1","timestamp":"2021-02-11T06:17:42.088459272Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TpoivCStluhaC0Hx2vRcWbz","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024262.113934_33.1","section":"bucket.instance","name":"xscaju-2:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.2","timestamp":"2021-02-11T06:17:42.113934461Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__hd20jVKBEurKa5-YHlmHCl","ver":1},"status":{"status":"write"}}},{"id":"1_1613024262.120087_34.1","section":"bucket.instance","name":"xscaju-2:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.2","timestamp":"2021-02-11T06:17:42.120087261Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__hd20jVKBEurKa5-YHlmHCl","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024262.122293_35.1","section":"bucket","name":"xscaju-2","timestamp":"2021-02-11T06:17:42.122293620Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XnfQzd3oXvAs9-JK2tnNiPr","ver":1},"status":{"status":"write"}}},{"id":"1_1613024262.127402_36.1","section":"bucket","name":"xscaju-2","timestamp":"2021-02-11T06:17:42.127402916Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XnfQzd3oXvAs9-JK2tnNiPr","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024265.332234_37.1","section":"bucket.instance","name":"xscaju-3:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.3","timestamp":"2021-02-11T06:17:45.332234271Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AWV7DltlSln-5qmsLkFyfua","ver":1},"status":{"status":"write"}}},{"id":"1_1613024265.350497_38.1","section":"bucket.instance","name":"xscaju-3:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.3","timestamp":"2021-02-11T06:17:45.350497485Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AWV7DltlSln-5qmsLkFyfua","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024265.352443_39.1","section":"bucket","name":"xscaju-3","timestamp":"2021-02-11T06:17:45.352443766Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wQHhrUn5k7F9mLuXRB8f1Xz","ver":1},"status":{"status":"write"}}},{"id":"1_1613024265.357008_40.1","section":"bucket","name":"xscaju-3","timestamp":"2021-02-11T06:17:45.357008799Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wQHhrUn5k7F9mLuXRB8f1Xz","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024279.487119_41.1","section":"bucket.instance","name":"xscaju-4:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.4","timestamp":"2021-02-11T06:17:59.487119032Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yaLqVGz15t2aAHd-7NPFL62","ver":1},"status":{"status":"write"}}},{"id":"1_1613024279.492264_43.1","section":"bucket.instance","name":"xscaju-4:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.4","timestamp":"2021-02-11T06:17:59.492264454Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yaLqVGz15t2aAHd-7NPFL62","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024279.494230_44.1","section":"bucket","name":"xscaju-4","timestamp":"2021-02-11T06:17:59.494230677Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_339NBHqlbS70ofKCpOAhJbQ","ver":1},"status":{"status":"write"}}},{"id":"1_1613024279.500216_45.1","section":"bucket","name":"xscaju-4","timestamp":"2021-02-11T06:17:59.500216604Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_339NBHqlbS70ofKCpOAhJbQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024279.522608_46.1","section":"bucket.instance","name":"xscaju-5:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.5","timestamp":"2021-02-11T06:17:59.522608088Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vkQG_Xp99-yJp-ToKtT8kg3","ver":1},"status":{"status":"write"}}},{"id":"1_1613024279.531162_47.1","section":"bucket.instance","name":"xscaju-5:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.5","timestamp":"2021-02-11T06:17:59.531162028Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vkQG_Xp99-yJp-ToKtT8kg3","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024279.533140_48.1","section":"bucket","name":"xscaju-5","timestamp":"2021-02-11T06:17:59.533140603Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SVeKSQ9gdskN7i7U-nYLEgi","ver":1},"status":{"status":"write"}}},{"id":"1_1613024279.540314_49.1","section":"bucket","name":"xscaju-5","timestamp":"2021-02-11T06:17:59.540314431Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SVeKSQ9gdskN7i7U-nYLEgi","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024279.587364_50.1","section":"bucket.instance","name":"xscaju-6:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.6","timestamp":"2021-02-11T06:17:59.587364825Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rte0LgEFVcWvYNPAx9nNs_n","ver":1},"status":{"status":"write"}}},{"id":"1_1613024279.595650_51.1","section":"bucket.instance","name":"xscaju-6:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.6","timestamp":"2021-02-11T06:17:59.595650557Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rte0LgEFVcWvYNPAx9nNs_n","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024279.599110_52.1","section":"bucket","name":"xscaju-6","timestamp":"2021-02-11T06:17:59.599110042Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9FvsuLFeId2rWrC78ZduVOd","ver":1},"status":{"status":"write"}}},{"id":"1_1613024279.610246_53.1","section":"bucket","name":"xscaju-6","timestamp":"2021-02-11T06:17:59.610246072Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9FvsuLFeId2rWrC78ZduVOd","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024294.216703_54.1","section":"bucket.instance","name":"xscaju-7:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.7","timestamp":"2021-02-11T06:18:14.216703877Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jWdNA4N3P_5daffu-K1YRGJ","ver":1},"status":{"status":"write"}}},{"id":"1_1613024294.222678_57.1","section":"bucket.instance","name":"xscaju-7:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.7","timestamp":"2021-02-11T06:18:14.222678059Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jWdNA4N3P_5daffu-K1YRGJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024294.224967_58.1","section":"bucket","name":"xscaju-7","timestamp":"2021-02-11T06:18:14.224967555Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_E45sVYzIU0DaBEmS8x7_Rdy","ver":1},"status":{"status":"write"}}},{"id":"1_1613024294.230021_59.1","section":"bucket","name":"xscaju-7","timestamp":"2021-02-11T06:18:14.230021669Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_E45sVYzIU0DaBEmS8x7_Rdy","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024294.249488_60.1","section":"bucket.instance","name":"xscaju-8:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.8","timestamp":"2021-02-11T06:18:14.249488401Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kfebcc_EHBe3uMLbBs7dDoL","ver":1},"status":{"status":"write"}}},{"id":"1_1613024294.255631_61.1","section":"bucket.instance","name":"xscaju-8:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.8","timestamp":"2021-02-11T06:18:14.255631956Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kfebcc_EHBe3uMLbBs7dDoL","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024294.257842_62.1","section":"bucket","name":"xscaju-8","timestamp":"2021-02-11T06:18:14.257842711Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Wh2l0tE943p3oAEfB-UgE1m","ver":1},"status":{"status":"write"}}},{"id":"1_1613024294.263232_63.1","section":"bucket","name":"xscaju-8","timestamp":"2021-02-11T06:18:14.263232235Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Wh2l0tE943p3oAEfB-UgE1m","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024294.307011_64.1","section":"bucket.instance","name":"xscaju-9:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.9","timestamp":"2021-02-11T06:18:14.307011641Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5js7vqwL9Cu1tRXnOerclkl","ver":1},"status":{"status":"write"}}},{"id":"1_1613024294.313905_65.1","section":"bucket.instance","name":"xscaju-9:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.9","timestamp":"2021-02-11T06:18:14.313905047Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5js7vqwL9Cu1tRXnOerclkl","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024294.315946_66.1","section":"bucket","name":"xscaju-9","timestamp":"2021-02-11T06:18:14.315946160Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UWAgQgKN2X4AJl_kFPBitiQ","ver":1},"status":{"status":"write"}}},{"id":"1_1613024294.322067_67.1","section":"bucket","name":"xscaju-9","timestamp":"2021-02-11T06:18:14.322067563Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UWAgQgKN2X4AJl_kFPBitiQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024300.918033_68.1","section":"bucket","name":"xscaju-7","timestamp":"2021-02-11T06:18:20.918033480Z","data":{"read_version":{"tag":"_E45sVYzIU0DaBEmS8x7_Rdy","ver":1},"write_version":{"tag":"_E45sVYzIU0DaBEmS8x7_Rdy","ver":2},"status":{"status":"remove"}}},{"id":"1_1613024300.924725_70.1","section":"bucket","name":"xscaju-7","timestamp":"2021-02-11T06:18:20.924725131Z","data":{"read_version":{"tag":"_E45sVYzIU0DaBEmS8x7_Rdy","ver":1},"write_version":{"tag":"_E45sVYzIU0DaBEmS8x7_Rdy","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024300.998791_71.1","section":"bucket","name":"xscaju-8","timestamp":"2021-02-11T06:18:20.998791991Z","data":{"read_version":{"tag":"_Wh2l0tE943p3oAEfB-UgE1m","ver":1},"write_version":{"tag":"_Wh2l0tE943p3oAEfB-UgE1m","ver":2},"status":{"status":"remove"}}},{"id":"1_1613024301.006963_72.1","section":"bucket","name":"xscaju-8","timestamp":"2021-02-11T06:18:21.006963296Z","data":{"read_version":{"tag":"_Wh2l0tE943p3oAEfB-UgE1m","ver":1},"write_version":{"tag":"_Wh2l0tE943p3oAEfB-UgE1m","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024301.135430_73.1","section":"bucket","name":"xscaju-9","timestamp":"2021-02-11T06:18:21.135430186Z","data":{"read_version":{"tag":"_UWAgQgKN2X4AJl_kFPBitiQ","ver":1},"write_version":{"tag":"_UWAgQgKN2X4AJl_kFPBitiQ","ver":2},"status":{"status":"remove"}}},{"id":"1_1613024301.142235_74.1","section":"bucket","name":"xscaju-9","timestamp":"2021-02-11T06:18:21.142235547Z","data":{"read_version":{"tag":"_UWAgQgKN2X4AJl_kFPBitiQ","ver":1},"write_version":{"tag":"_UWAgQgKN2X4AJl_kFPBitiQ","ver":2},"status":{"status":"complete"}}}] 2021-02-11T06:28:38.514 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '7c4a812f-81e4-4a72-acce-0ee18589648c', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:28:38.514 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '7c4a812f-81e4-4a72-acce-0ee18589648c', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:28:38.514 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 7c4a812f-81e4-4a72-acce-0ee18589648c --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:28:38.553 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:28:38.553 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:28:38.814 INFO:teuthology.orchestra.run.smithi172.stdout:[{"id":"1_1613024135.173475_43.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2021-02-11T06:15:35.173475237Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SbNMO4uv9J_5-KcMlPHXiJf","ver":1},"status":{"status":"write"}}},{"id":"1_1613024135.181851_48.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2021-02-11T06:15:35.181851493Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SbNMO4uv9J_5-KcMlPHXiJf","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024304.838286_55.1","section":"bucket.instance","name":"xscaju-10:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.10","timestamp":"2021-02-11T06:18:24.838286662Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vUDtGJ3GBQdG6kVLldnqDId","ver":1},"status":{"status":"write"}}},{"id":"1_1613024304.844317_62.1","section":"bucket.instance","name":"xscaju-10:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.10","timestamp":"2021-02-11T06:18:24.844317783Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vUDtGJ3GBQdG6kVLldnqDId","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024304.846426_63.1","section":"bucket","name":"xscaju-10","timestamp":"2021-02-11T06:18:24.846426466Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pfKfSMc5TvrJjKRiccmEVID","ver":1},"status":{"status":"write"}}},{"id":"1_1613024304.852830_64.1","section":"bucket","name":"xscaju-10","timestamp":"2021-02-11T06:18:24.852830013Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pfKfSMc5TvrJjKRiccmEVID","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024304.876275_65.1","section":"bucket.instance","name":"xscaju-11:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.11","timestamp":"2021-02-11T06:18:24.876275792Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7O37Ypf5_pArN3rvyGMahZQ","ver":1},"status":{"status":"write"}}},{"id":"1_1613024304.882604_66.1","section":"bucket.instance","name":"xscaju-11:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.11","timestamp":"2021-02-11T06:18:24.882604435Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7O37Ypf5_pArN3rvyGMahZQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024304.884842_67.1","section":"bucket","name":"xscaju-11","timestamp":"2021-02-11T06:18:24.884842370Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hC41ht1E97pbsgCJUm_8lj_","ver":1},"status":{"status":"write"}}},{"id":"1_1613024304.890580_68.1","section":"bucket","name":"xscaju-11","timestamp":"2021-02-11T06:18:24.890580142Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hC41ht1E97pbsgCJUm_8lj_","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024304.936648_69.1","section":"bucket.instance","name":"xscaju-12:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.12","timestamp":"2021-02-11T06:18:24.936648345Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rdqYRx6VnHnoYGod-A0qeUL","ver":1},"status":{"status":"write"}}},{"id":"1_1613024304.943318_70.1","section":"bucket.instance","name":"xscaju-12:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.12","timestamp":"2021-02-11T06:18:24.943318666Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rdqYRx6VnHnoYGod-A0qeUL","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024304.946295_71.1","section":"bucket","name":"xscaju-12","timestamp":"2021-02-11T06:18:24.946295639Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YM8ZNKbVwbAQ1oc9Cz4_OM_","ver":1},"status":{"status":"write"}}},{"id":"1_1613024304.951770_72.1","section":"bucket","name":"xscaju-12","timestamp":"2021-02-11T06:18:24.951770048Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YM8ZNKbVwbAQ1oc9Cz4_OM_","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024341.501587_78.1","section":"bucket.instance","name":"xscaju-13:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.13","timestamp":"2021-02-11T06:19:01.501587544Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BCFlY5FQTgcsjiMtHT5pskt","ver":1},"status":{"status":"write"}}},{"id":"1_1613024341.507190_81.1","section":"bucket.instance","name":"xscaju-13:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.13","timestamp":"2021-02-11T06:19:01.507190496Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BCFlY5FQTgcsjiMtHT5pskt","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024341.509363_82.1","section":"bucket","name":"xscaju-13","timestamp":"2021-02-11T06:19:01.509363808Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_i2XT5Un3mY87cqRlmnmKTE1","ver":1},"status":{"status":"write"}}},{"id":"1_1613024341.515079_83.1","section":"bucket","name":"xscaju-13","timestamp":"2021-02-11T06:19:01.515079048Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_i2XT5Un3mY87cqRlmnmKTE1","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024341.540023_84.1","section":"bucket.instance","name":"xscaju-14:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.14","timestamp":"2021-02-11T06:19:01.540023533Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OG9OQa6kjk0iKLLVPQDaiG-","ver":1},"status":{"status":"write"}}},{"id":"1_1613024341.546055_85.1","section":"bucket.instance","name":"xscaju-14:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.14","timestamp":"2021-02-11T06:19:01.546055113Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OG9OQa6kjk0iKLLVPQDaiG-","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024341.548182_86.1","section":"bucket","name":"xscaju-14","timestamp":"2021-02-11T06:19:01.548182976Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LolNl85pkAeXvxWPRnXMOqS","ver":1},"status":{"status":"write"}}},{"id":"1_1613024341.553747_87.1","section":"bucket","name":"xscaju-14","timestamp":"2021-02-11T06:19:01.553747682Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LolNl85pkAeXvxWPRnXMOqS","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024341.605553_88.1","section":"bucket.instance","name":"xscaju-15:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.15","timestamp":"2021-02-11T06:19:01.605553336Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UrhgCwLCqd88P1DuEAbHlw5","ver":1},"status":{"status":"write"}}},{"id":"1_1613024341.611354_89.1","section":"bucket.instance","name":"xscaju-15:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.15","timestamp":"2021-02-11T06:19:01.611354865Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UrhgCwLCqd88P1DuEAbHlw5","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024341.613945_90.1","section":"bucket","name":"xscaju-15","timestamp":"2021-02-11T06:19:01.613945279Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FYxrvNRyhtUMgtpEdrH0XpH","ver":1},"status":{"status":"write"}}},{"id":"1_1613024341.620645_91.1","section":"bucket","name":"xscaju-15","timestamp":"2021-02-11T06:19:01.620645045Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FYxrvNRyhtUMgtpEdrH0XpH","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024373.488563_97.1","section":"bucket.instance","name":"xscaju-16:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.16","timestamp":"2021-02-11T06:19:33.488563051Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rRbEM8RJEpDuvs5eWMQtblo","ver":1},"status":{"status":"write"}}},{"id":"1_1613024373.493847_99.1","section":"bucket.instance","name":"xscaju-16:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.16","timestamp":"2021-02-11T06:19:33.493847401Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rRbEM8RJEpDuvs5eWMQtblo","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024373.495976_100.1","section":"bucket","name":"xscaju-16","timestamp":"2021-02-11T06:19:33.495976000Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_sHQb-O2kysovE_jM77orOZM","ver":1},"status":{"status":"write"}}},{"id":"1_1613024373.501119_101.1","section":"bucket","name":"xscaju-16","timestamp":"2021-02-11T06:19:33.501119742Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_sHQb-O2kysovE_jM77orOZM","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024373.524093_102.1","section":"bucket.instance","name":"xscaju-17:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.17","timestamp":"2021-02-11T06:19:33.524093914Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__tpF2cPNLKiB6VVx7L7bGVh","ver":1},"status":{"status":"write"}}},{"id":"1_1613024373.529677_103.1","section":"bucket.instance","name":"xscaju-17:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.17","timestamp":"2021-02-11T06:19:33.529677757Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__tpF2cPNLKiB6VVx7L7bGVh","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024373.531628_104.1","section":"bucket","name":"xscaju-17","timestamp":"2021-02-11T06:19:33.531628576Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_EbbwHqhEqjL5axWisAmsO1p","ver":1},"status":{"status":"write"}}},{"id":"1_1613024373.538228_105.1","section":"bucket","name":"xscaju-17","timestamp":"2021-02-11T06:19:33.538228660Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_EbbwHqhEqjL5axWisAmsO1p","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024373.588815_106.1","section":"bucket.instance","name":"xscaju-18:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.18","timestamp":"2021-02-11T06:19:33.588815655Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1WW2O12ltbkT25Ghnp053dJ","ver":1},"status":{"status":"write"}}},{"id":"1_1613024373.596722_107.1","section":"bucket.instance","name":"xscaju-18:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.18","timestamp":"2021-02-11T06:19:33.596722655Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1WW2O12ltbkT25Ghnp053dJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024373.598875_108.1","section":"bucket","name":"xscaju-18","timestamp":"2021-02-11T06:19:33.598875179Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XEjzOMy0gl9N4M1Va8lXg9Y","ver":1},"status":{"status":"write"}}},{"id":"1_1613024373.606297_109.1","section":"bucket","name":"xscaju-18","timestamp":"2021-02-11T06:19:33.606297952Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XEjzOMy0gl9N4M1Va8lXg9Y","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024373.634176_110.1","section":"bucket.instance","name":"xscaju-16:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.16","timestamp":"2021-02-11T06:19:33.634176115Z","data":{"read_version":{"tag":"_rRbEM8RJEpDuvs5eWMQtblo","ver":1},"write_version":{"tag":"_rRbEM8RJEpDuvs5eWMQtblo","ver":2},"status":{"status":"write"}}},{"id":"1_1613024373.640813_111.1","section":"bucket.instance","name":"xscaju-16:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.16","timestamp":"2021-02-11T06:19:33.640813726Z","data":{"read_version":{"tag":"_rRbEM8RJEpDuvs5eWMQtblo","ver":1},"write_version":{"tag":"_rRbEM8RJEpDuvs5eWMQtblo","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024373.646623_112.1","section":"bucket.instance","name":"xscaju-17:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.17","timestamp":"2021-02-11T06:19:33.646623596Z","data":{"read_version":{"tag":"__tpF2cPNLKiB6VVx7L7bGVh","ver":1},"write_version":{"tag":"__tpF2cPNLKiB6VVx7L7bGVh","ver":2},"status":{"status":"write"}}},{"id":"1_1613024373.663550_113.1","section":"bucket.instance","name":"xscaju-17:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.17","timestamp":"2021-02-11T06:19:33.663550101Z","data":{"read_version":{"tag":"__tpF2cPNLKiB6VVx7L7bGVh","ver":1},"write_version":{"tag":"__tpF2cPNLKiB6VVx7L7bGVh","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024373.683136_114.1","section":"bucket.instance","name":"xscaju-18:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.18","timestamp":"2021-02-11T06:19:33.683136588Z","data":{"read_version":{"tag":"_1WW2O12ltbkT25Ghnp053dJ","ver":1},"write_version":{"tag":"_1WW2O12ltbkT25Ghnp053dJ","ver":2},"status":{"status":"write"}}},{"id":"1_1613024373.689054_115.1","section":"bucket.instance","name":"xscaju-18:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.18","timestamp":"2021-02-11T06:19:33.689054244Z","data":{"read_version":{"tag":"_1WW2O12ltbkT25Ghnp053dJ","ver":1},"write_version":{"tag":"_1WW2O12ltbkT25Ghnp053dJ","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024511.261879_127.1","section":"bucket.instance","name":"xscaju-19:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.19","timestamp":"2021-02-11T06:21:51.261879107Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_p_eBM2glYgyD_bvacpJjzWg","ver":1},"status":{"status":"write"}}},{"id":"1_1613024511.267174_134.1","section":"bucket.instance","name":"xscaju-19:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.19","timestamp":"2021-02-11T06:21:51.267174934Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_p_eBM2glYgyD_bvacpJjzWg","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024511.268990_135.1","section":"bucket","name":"xscaju-19","timestamp":"2021-02-11T06:21:51.268990551Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KmgBNF2u5Y56huMJaZaInaQ","ver":1},"status":{"status":"write"}}},{"id":"1_1613024511.274597_136.1","section":"bucket","name":"xscaju-19","timestamp":"2021-02-11T06:21:51.274597146Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KmgBNF2u5Y56huMJaZaInaQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024511.281125_137.1","section":"bucket.instance","name":"xscaju-19:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.19","timestamp":"2021-02-11T06:21:51.281125877Z","data":{"read_version":{"tag":"_p_eBM2glYgyD_bvacpJjzWg","ver":1},"write_version":{"tag":"_p_eBM2glYgyD_bvacpJjzWg","ver":2},"status":{"status":"write"}}},{"id":"1_1613024511.287525_138.1","section":"bucket.instance","name":"xscaju-19:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.19","timestamp":"2021-02-11T06:21:51.287525217Z","data":{"read_version":{"tag":"_p_eBM2glYgyD_bvacpJjzWg","ver":1},"write_version":{"tag":"_p_eBM2glYgyD_bvacpJjzWg","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024655.788162_148.1","section":"bucket.instance","name":"xscaju-20:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.20","timestamp":"2021-02-11T06:24:15.788162830Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-CKkQFoycEeUKncKE438im2","ver":1},"status":{"status":"write"}}},{"id":"1_1613024655.793777_154.1","section":"bucket.instance","name":"xscaju-20:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.20","timestamp":"2021-02-11T06:24:15.793777725Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-CKkQFoycEeUKncKE438im2","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024655.795702_155.1","section":"bucket","name":"xscaju-20","timestamp":"2021-02-11T06:24:15.795702477Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RLExy4tiUpASD0qI4I42NP-","ver":1},"status":{"status":"write"}}},{"id":"1_1613024655.802479_156.1","section":"bucket","name":"xscaju-20","timestamp":"2021-02-11T06:24:15.802479510Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RLExy4tiUpASD0qI4I42NP-","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024660.020192_159.1","section":"bucket.instance","name":"xscaju-20:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.20","timestamp":"2021-02-11T06:24:20.020192845Z","data":{"read_version":{"tag":"_-CKkQFoycEeUKncKE438im2","ver":1},"write_version":{"tag":"_-CKkQFoycEeUKncKE438im2","ver":2},"status":{"status":"write"}}},{"id":"1_1613024660.030516_162.1","section":"bucket.instance","name":"xscaju-20:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.20","timestamp":"2021-02-11T06:24:20.030516613Z","data":{"read_version":{"tag":"_-CKkQFoycEeUKncKE438im2","ver":1},"write_version":{"tag":"_-CKkQFoycEeUKncKE438im2","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024692.032178_166.1","section":"bucket.instance","name":"xscaju-20:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.20","timestamp":"2021-02-11T06:24:52.032178508Z","data":{"read_version":{"tag":"_-CKkQFoycEeUKncKE438im2","ver":2},"write_version":{"tag":"_-CKkQFoycEeUKncKE438im2","ver":3},"status":{"status":"write"}}},{"id":"1_1613024692.042180_169.1","section":"bucket.instance","name":"xscaju-20:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.20","timestamp":"2021-02-11T06:24:52.042180753Z","data":{"read_version":{"tag":"_-CKkQFoycEeUKncKE438im2","ver":2},"write_version":{"tag":"_-CKkQFoycEeUKncKE438im2","ver":3},"status":{"status":"complete"}}},{"id":"1_1613024712.687247_172.1","section":"bucket.instance","name":"xscaju-21:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.21","timestamp":"2021-02-11T06:25:12.687247976Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_aghkHsnfiSBGUtPpitq6QYX","ver":1},"status":{"status":"write"}}},{"id":"1_1613024712.692691_175.1","section":"bucket.instance","name":"xscaju-21:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.21","timestamp":"2021-02-11T06:25:12.692691830Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_aghkHsnfiSBGUtPpitq6QYX","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024712.694732_176.1","section":"bucket","name":"xscaju-21","timestamp":"2021-02-11T06:25:12.694732388Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dEuDzfRlfV9QmLUwaDaXb1v","ver":1},"status":{"status":"write"}}},{"id":"1_1613024712.701991_177.1","section":"bucket","name":"xscaju-21","timestamp":"2021-02-11T06:25:12.701991779Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dEuDzfRlfV9QmLUwaDaXb1v","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024712.730036_178.1","section":"bucket.instance","name":"xscaju-22:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.22","timestamp":"2021-02-11T06:25:12.730036078Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9xhfxddv4lQiZriVKjsuyUG","ver":1},"status":{"status":"write"}}},{"id":"1_1613024712.735883_179.1","section":"bucket.instance","name":"xscaju-22:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.22","timestamp":"2021-02-11T06:25:12.735883165Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9xhfxddv4lQiZriVKjsuyUG","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024712.737970_180.1","section":"bucket","name":"xscaju-22","timestamp":"2021-02-11T06:25:12.737970149Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VmQT_CTFegH5L6LSM_iEfbk","ver":1},"status":{"status":"write"}}},{"id":"1_1613024712.744137_181.1","section":"bucket","name":"xscaju-22","timestamp":"2021-02-11T06:25:12.744137116Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VmQT_CTFegH5L6LSM_iEfbk","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024712.790840_182.1","section":"bucket.instance","name":"xscaju-23:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.23","timestamp":"2021-02-11T06:25:12.790840443Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_EVe05TdaPvo0mOj_E-9Ws5m","ver":1},"status":{"status":"write"}}},{"id":"1_1613024712.796616_183.1","section":"bucket.instance","name":"xscaju-23:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.23","timestamp":"2021-02-11T06:25:12.796616431Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_EVe05TdaPvo0mOj_E-9Ws5m","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024712.798717_184.1","section":"bucket","name":"xscaju-23","timestamp":"2021-02-11T06:25:12.798717915Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rk7ucCpCL0fx3vjZKmGG63K","ver":1},"status":{"status":"write"}}},{"id":"1_1613024712.805163_185.1","section":"bucket","name":"xscaju-23","timestamp":"2021-02-11T06:25:12.805163379Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rk7ucCpCL0fx3vjZKmGG63K","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024712.844997_186.1","section":"bucket.instance","name":"xscaju-21:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.21","timestamp":"2021-02-11T06:25:12.844997230Z","data":{"read_version":{"tag":"_aghkHsnfiSBGUtPpitq6QYX","ver":1},"write_version":{"tag":"_aghkHsnfiSBGUtPpitq6QYX","ver":2},"status":{"status":"write"}}},{"id":"1_1613024712.852513_187.1","section":"bucket.instance","name":"xscaju-21:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.21","timestamp":"2021-02-11T06:25:12.852513487Z","data":{"read_version":{"tag":"_aghkHsnfiSBGUtPpitq6QYX","ver":1},"write_version":{"tag":"_aghkHsnfiSBGUtPpitq6QYX","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024712.860311_188.1","section":"bucket.instance","name":"xscaju-22:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.22","timestamp":"2021-02-11T06:25:12.860311320Z","data":{"read_version":{"tag":"_9xhfxddv4lQiZriVKjsuyUG","ver":1},"write_version":{"tag":"_9xhfxddv4lQiZriVKjsuyUG","ver":2},"status":{"status":"write"}}},{"id":"1_1613024712.875536_189.1","section":"bucket.instance","name":"xscaju-22:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.22","timestamp":"2021-02-11T06:25:12.875536257Z","data":{"read_version":{"tag":"_9xhfxddv4lQiZriVKjsuyUG","ver":1},"write_version":{"tag":"_9xhfxddv4lQiZriVKjsuyUG","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024712.887831_190.1","section":"bucket.instance","name":"xscaju-23:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.23","timestamp":"2021-02-11T06:25:12.887831436Z","data":{"read_version":{"tag":"_EVe05TdaPvo0mOj_E-9Ws5m","ver":1},"write_version":{"tag":"_EVe05TdaPvo0mOj_E-9Ws5m","ver":2},"status":{"status":"write"}}},{"id":"1_1613024712.893011_191.1","section":"bucket.instance","name":"xscaju-23:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.23","timestamp":"2021-02-11T06:25:12.893011226Z","data":{"read_version":{"tag":"_EVe05TdaPvo0mOj_E-9Ws5m","ver":1},"write_version":{"tag":"_EVe05TdaPvo0mOj_E-9Ws5m","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024730.813367_209.1","section":"bucket.instance","name":"xscaju-24:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.24","timestamp":"2021-02-11T06:25:30.813367288Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fIGyWcOWpuc4VeT86y9JIXc","ver":1},"status":{"status":"write"}}},{"id":"1_1613024730.818704_214.1","section":"bucket.instance","name":"xscaju-24:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.24","timestamp":"2021-02-11T06:25:30.818704273Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fIGyWcOWpuc4VeT86y9JIXc","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024730.820590_215.1","section":"bucket","name":"xscaju-24","timestamp":"2021-02-11T06:25:30.820590878Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CdxZUOujdzmSeUwguD2f3ih","ver":1},"status":{"status":"write"}}},{"id":"1_1613024730.828498_216.1","section":"bucket","name":"xscaju-24","timestamp":"2021-02-11T06:25:30.828498037Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CdxZUOujdzmSeUwguD2f3ih","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024730.853747_217.1","section":"bucket.instance","name":"xscaju-25:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.25","timestamp":"2021-02-11T06:25:30.853747007Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1wwb-pQJBpU2ZVKZ9xscYjF","ver":1},"status":{"status":"write"}}},{"id":"1_1613024730.860417_218.1","section":"bucket.instance","name":"xscaju-25:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.25","timestamp":"2021-02-11T06:25:30.860417124Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1wwb-pQJBpU2ZVKZ9xscYjF","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024730.862570_219.1","section":"bucket","name":"xscaju-25","timestamp":"2021-02-11T06:25:30.862570482Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RO4pHUixO0cYr5mTSfAoRR2","ver":1},"status":{"status":"write"}}},{"id":"1_1613024730.869244_220.1","section":"bucket","name":"xscaju-25","timestamp":"2021-02-11T06:25:30.869244273Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RO4pHUixO0cYr5mTSfAoRR2","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024730.918298_221.1","section":"bucket.instance","name":"xscaju-26:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.26","timestamp":"2021-02-11T06:25:30.918298568Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ztmpV4a7sa_Si3ePxy8pHiQ","ver":1},"status":{"status":"write"}}},{"id":"1_1613024730.924750_222.1","section":"bucket.instance","name":"xscaju-26:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.26","timestamp":"2021-02-11T06:25:30.924750148Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ztmpV4a7sa_Si3ePxy8pHiQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024730.927008_223.1","section":"bucket","name":"xscaju-26","timestamp":"2021-02-11T06:25:30.927008043Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5hf0n5H6P98HZy7pnWOkiJS","ver":1},"status":{"status":"write"}}},{"id":"1_1613024730.933901_224.1","section":"bucket","name":"xscaju-26","timestamp":"2021-02-11T06:25:30.933901776Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5hf0n5H6P98HZy7pnWOkiJS","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024730.959079_225.1","section":"bucket.instance","name":"xscaju-24:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.24","timestamp":"2021-02-11T06:25:30.959079377Z","data":{"read_version":{"tag":"_fIGyWcOWpuc4VeT86y9JIXc","ver":1},"write_version":{"tag":"_fIGyWcOWpuc4VeT86y9JIXc","ver":2},"status":{"status":"write"}}},{"id":"1_1613024730.965401_226.1","section":"bucket.instance","name":"xscaju-24:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.24","timestamp":"2021-02-11T06:25:30.965401947Z","data":{"read_version":{"tag":"_fIGyWcOWpuc4VeT86y9JIXc","ver":1},"write_version":{"tag":"_fIGyWcOWpuc4VeT86y9JIXc","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024730.969631_227.1","section":"bucket.instance","name":"xscaju-24:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.24","timestamp":"2021-02-11T06:25:30.969631892Z","data":{"read_version":{"tag":"_fIGyWcOWpuc4VeT86y9JIXc","ver":2},"write_version":{"tag":"_fIGyWcOWpuc4VeT86y9JIXc","ver":3},"status":{"status":"write"}}},{"id":"1_1613024730.977493_228.1","section":"bucket.instance","name":"xscaju-24:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.24","timestamp":"2021-02-11T06:25:30.977493784Z","data":{"read_version":{"tag":"_fIGyWcOWpuc4VeT86y9JIXc","ver":2},"write_version":{"tag":"_fIGyWcOWpuc4VeT86y9JIXc","ver":3},"status":{"status":"complete"}}},{"id":"1_1613024730.982502_229.1","section":"bucket.instance","name":"xscaju-25:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.25","timestamp":"2021-02-11T06:25:30.982502378Z","data":{"read_version":{"tag":"_1wwb-pQJBpU2ZVKZ9xscYjF","ver":1},"write_version":{"tag":"_1wwb-pQJBpU2ZVKZ9xscYjF","ver":2},"status":{"status":"write"}}},{"id":"1_1613024730.988537_230.1","section":"bucket.instance","name":"xscaju-25:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.25","timestamp":"2021-02-11T06:25:30.988537741Z","data":{"read_version":{"tag":"_1wwb-pQJBpU2ZVKZ9xscYjF","ver":1},"write_version":{"tag":"_1wwb-pQJBpU2ZVKZ9xscYjF","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024731.000808_231.1","section":"bucket.instance","name":"xscaju-25:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.25","timestamp":"2021-02-11T06:25:31.000808166Z","data":{"read_version":{"tag":"_1wwb-pQJBpU2ZVKZ9xscYjF","ver":2},"write_version":{"tag":"_1wwb-pQJBpU2ZVKZ9xscYjF","ver":3},"status":{"status":"write"}}},{"id":"1_1613024731.008870_232.1","section":"bucket.instance","name":"xscaju-25:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.25","timestamp":"2021-02-11T06:25:31.008870886Z","data":{"read_version":{"tag":"_1wwb-pQJBpU2ZVKZ9xscYjF","ver":2},"write_version":{"tag":"_1wwb-pQJBpU2ZVKZ9xscYjF","ver":3},"status":{"status":"complete"}}},{"id":"1_1613024731.023118_233.1","section":"bucket.instance","name":"xscaju-26:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.26","timestamp":"2021-02-11T06:25:31.023118052Z","data":{"read_version":{"tag":"_ztmpV4a7sa_Si3ePxy8pHiQ","ver":1},"write_version":{"tag":"_ztmpV4a7sa_Si3ePxy8pHiQ","ver":2},"status":{"status":"write"}}},{"id":"1_1613024731.047850_234.1","section":"bucket.instance","name":"xscaju-26:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.26","timestamp":"2021-02-11T06:25:31.047850325Z","data":{"read_version":{"tag":"_ztmpV4a7sa_Si3ePxy8pHiQ","ver":1},"write_version":{"tag":"_ztmpV4a7sa_Si3ePxy8pHiQ","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024731.065201_235.1","section":"bucket.instance","name":"xscaju-26:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.26","timestamp":"2021-02-11T06:25:31.065201447Z","data":{"read_version":{"tag":"_ztmpV4a7sa_Si3ePxy8pHiQ","ver":2},"write_version":{"tag":"_ztmpV4a7sa_Si3ePxy8pHiQ","ver":3},"status":{"status":"write"}}},{"id":"1_1613024731.075841_236.1","section":"bucket.instance","name":"xscaju-26:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.26","timestamp":"2021-02-11T06:25:31.075841634Z","data":{"read_version":{"tag":"_ztmpV4a7sa_Si3ePxy8pHiQ","ver":2},"write_version":{"tag":"_ztmpV4a7sa_Si3ePxy8pHiQ","ver":3},"status":{"status":"complete"}}},{"id":"1_1613024740.384811_241.1","section":"bucket.instance","name":"xscaju-27:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.27","timestamp":"2021-02-11T06:25:40.384811683Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TrIDSZS_H7CX19SiRjfTfaj","ver":1},"status":{"status":"write"}}},{"id":"1_1613024740.391229_248.1","section":"bucket.instance","name":"xscaju-27:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.27","timestamp":"2021-02-11T06:25:40.391229729Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TrIDSZS_H7CX19SiRjfTfaj","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024740.393342_249.1","section":"bucket","name":"xscaju-27","timestamp":"2021-02-11T06:25:40.393342210Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MTFcHaBN2BPut8Vpf8NS_O4","ver":1},"status":{"status":"write"}}},{"id":"1_1613024740.400053_250.1","section":"bucket","name":"xscaju-27","timestamp":"2021-02-11T06:25:40.400053238Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MTFcHaBN2BPut8Vpf8NS_O4","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024740.431618_251.1","section":"bucket.instance","name":"xscaju-28:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.28","timestamp":"2021-02-11T06:25:40.431618192Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_E6YSJ7lYgxp56BC7g_Mr5bI","ver":1},"status":{"status":"write"}}},{"id":"1_1613024740.439180_252.1","section":"bucket.instance","name":"xscaju-28:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.28","timestamp":"2021-02-11T06:25:40.439180036Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_E6YSJ7lYgxp56BC7g_Mr5bI","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024740.441086_253.1","section":"bucket","name":"xscaju-28","timestamp":"2021-02-11T06:25:40.441086013Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CCfBuLGioQost5PsQ4vGmNg","ver":1},"status":{"status":"write"}}},{"id":"1_1613024740.446195_254.1","section":"bucket","name":"xscaju-28","timestamp":"2021-02-11T06:25:40.446195665Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CCfBuLGioQost5PsQ4vGmNg","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024740.493564_255.1","section":"bucket.instance","name":"xscaju-29:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.29","timestamp":"2021-02-11T06:25:40.493564496Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Yn0qV-p0U41AFIUolLY2mOO","ver":1},"status":{"status":"write"}}},{"id":"1_1613024740.499171_256.1","section":"bucket.instance","name":"xscaju-29:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.29","timestamp":"2021-02-11T06:25:40.499171383Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Yn0qV-p0U41AFIUolLY2mOO","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024740.501027_257.1","section":"bucket","name":"xscaju-29","timestamp":"2021-02-11T06:25:40.501027370Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GbXKtKJm-8W6BT_rbT9R-d5","ver":1},"status":{"status":"write"}}},{"id":"1_1613024740.505530_258.1","section":"bucket","name":"xscaju-29","timestamp":"2021-02-11T06:25:40.505530321Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GbXKtKJm-8W6BT_rbT9R-d5","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024740.528269_259.1","section":"bucket.instance","name":"xscaju-27:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.27","timestamp":"2021-02-11T06:25:40.528269173Z","data":{"read_version":{"tag":"_TrIDSZS_H7CX19SiRjfTfaj","ver":1},"write_version":{"tag":"_TrIDSZS_H7CX19SiRjfTfaj","ver":2},"status":{"status":"write"}}},{"id":"1_1613024740.536371_260.1","section":"bucket.instance","name":"xscaju-27:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.27","timestamp":"2021-02-11T06:25:40.536371598Z","data":{"read_version":{"tag":"_TrIDSZS_H7CX19SiRjfTfaj","ver":1},"write_version":{"tag":"_TrIDSZS_H7CX19SiRjfTfaj","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024740.587862_261.1","section":"bucket.instance","name":"xscaju-28:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.28","timestamp":"2021-02-11T06:25:40.587862611Z","data":{"read_version":{"tag":"_E6YSJ7lYgxp56BC7g_Mr5bI","ver":1},"write_version":{"tag":"_E6YSJ7lYgxp56BC7g_Mr5bI","ver":2},"status":{"status":"write"}}},{"id":"1_1613024740.598686_262.1","section":"bucket.instance","name":"xscaju-28:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.28","timestamp":"2021-02-11T06:25:40.598686969Z","data":{"read_version":{"tag":"_E6YSJ7lYgxp56BC7g_Mr5bI","ver":1},"write_version":{"tag":"_E6YSJ7lYgxp56BC7g_Mr5bI","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024740.657924_263.1","section":"bucket.instance","name":"xscaju-29:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.29","timestamp":"2021-02-11T06:25:40.657924925Z","data":{"read_version":{"tag":"_Yn0qV-p0U41AFIUolLY2mOO","ver":1},"write_version":{"tag":"_Yn0qV-p0U41AFIUolLY2mOO","ver":2},"status":{"status":"write"}}},{"id":"1_1613024740.663844_264.1","section":"bucket.instance","name":"xscaju-29:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.29","timestamp":"2021-02-11T06:25:40.663844562Z","data":{"read_version":{"tag":"_Yn0qV-p0U41AFIUolLY2mOO","ver":1},"write_version":{"tag":"_Yn0qV-p0U41AFIUolLY2mOO","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024740.732516_265.1","section":"bucket.instance","name":"xscaju-30:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.30","timestamp":"2021-02-11T06:25:40.732516256Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MpzOxgSOKBraax9y0VvUydp","ver":1},"status":{"status":"write"}}},{"id":"1_1613024740.738207_266.1","section":"bucket.instance","name":"xscaju-30:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.30","timestamp":"2021-02-11T06:25:40.738207518Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MpzOxgSOKBraax9y0VvUydp","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024740.743036_267.1","section":"bucket","name":"xscaju-30","timestamp":"2021-02-11T06:25:40.743036899Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lLyPeKn1R0RlJIRU_JnNy5C","ver":1},"status":{"status":"write"}}},{"id":"1_1613024740.748807_268.1","section":"bucket","name":"xscaju-30","timestamp":"2021-02-11T06:25:40.748807837Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lLyPeKn1R0RlJIRU_JnNy5C","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024740.778162_269.1","section":"bucket.instance","name":"xscaju-31:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.31","timestamp":"2021-02-11T06:25:40.778162307Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_B7YFWyoUW2C3eKpdBi1oTca","ver":1},"status":{"status":"write"}}},{"id":"1_1613024740.784274_270.1","section":"bucket.instance","name":"xscaju-31:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.31","timestamp":"2021-02-11T06:25:40.784274697Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_B7YFWyoUW2C3eKpdBi1oTca","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024740.786489_271.1","section":"bucket","name":"xscaju-31","timestamp":"2021-02-11T06:25:40.786489748Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6CgMvFY80o--FV0KBLIcoKv","ver":1},"status":{"status":"write"}}},{"id":"1_1613024740.792101_272.1","section":"bucket","name":"xscaju-31","timestamp":"2021-02-11T06:25:40.792101132Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6CgMvFY80o--FV0KBLIcoKv","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024740.839308_273.1","section":"bucket.instance","name":"xscaju-32:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.32","timestamp":"2021-02-11T06:25:40.839308348Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8AE67v6AOAFpGgM2PS34g7g","ver":1},"status":{"status":"write"}}},{"id":"1_1613024740.851777_274.1","section":"bucket.instance","name":"xscaju-32:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.32","timestamp":"2021-02-11T06:25:40.851777607Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8AE67v6AOAFpGgM2PS34g7g","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024740.854155_275.1","section":"bucket","name":"xscaju-32","timestamp":"2021-02-11T06:25:40.854155749Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2Wa214TB-anBV6UIn3SPc9R","ver":1},"status":{"status":"write"}}},{"id":"1_1613024740.858532_276.1","section":"bucket","name":"xscaju-32","timestamp":"2021-02-11T06:25:40.858532323Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2Wa214TB-anBV6UIn3SPc9R","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024740.885186_277.1","section":"bucket.instance","name":"xscaju-30:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.30","timestamp":"2021-02-11T06:25:40.885186525Z","data":{"read_version":{"tag":"_MpzOxgSOKBraax9y0VvUydp","ver":1},"write_version":{"tag":"_MpzOxgSOKBraax9y0VvUydp","ver":2},"status":{"status":"write"}}},{"id":"1_1613024740.890116_278.1","section":"bucket.instance","name":"xscaju-30:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.30","timestamp":"2021-02-11T06:25:40.890116133Z","data":{"read_version":{"tag":"_MpzOxgSOKBraax9y0VvUydp","ver":1},"write_version":{"tag":"_MpzOxgSOKBraax9y0VvUydp","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024740.941521_279.1","section":"bucket.instance","name":"xscaju-31:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.31","timestamp":"2021-02-11T06:25:40.941521555Z","data":{"read_version":{"tag":"_B7YFWyoUW2C3eKpdBi1oTca","ver":1},"write_version":{"tag":"_B7YFWyoUW2C3eKpdBi1oTca","ver":2},"status":{"status":"write"}}},{"id":"1_1613024740.951603_280.1","section":"bucket.instance","name":"xscaju-31:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.31","timestamp":"2021-02-11T06:25:40.951603871Z","data":{"read_version":{"tag":"_B7YFWyoUW2C3eKpdBi1oTca","ver":1},"write_version":{"tag":"_B7YFWyoUW2C3eKpdBi1oTca","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024741.009272_281.1","section":"bucket.instance","name":"xscaju-32:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.32","timestamp":"2021-02-11T06:25:41.009272905Z","data":{"read_version":{"tag":"_8AE67v6AOAFpGgM2PS34g7g","ver":1},"write_version":{"tag":"_8AE67v6AOAFpGgM2PS34g7g","ver":2},"status":{"status":"write"}}},{"id":"1_1613024741.020533_282.1","section":"bucket.instance","name":"xscaju-32:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.32","timestamp":"2021-02-11T06:25:41.020533268Z","data":{"read_version":{"tag":"_8AE67v6AOAFpGgM2PS34g7g","ver":1},"write_version":{"tag":"_8AE67v6AOAFpGgM2PS34g7g","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024741.093044_283.1","section":"bucket.instance","name":"xscaju-33:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.33","timestamp":"2021-02-11T06:25:41.093044505Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Y1XZdFxjTuQdDNeNiv617DK","ver":1},"status":{"status":"write"}}},{"id":"1_1613024741.098818_284.1","section":"bucket.instance","name":"xscaju-33:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.33","timestamp":"2021-02-11T06:25:41.098818850Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Y1XZdFxjTuQdDNeNiv617DK","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024741.100781_285.1","section":"bucket","name":"xscaju-33","timestamp":"2021-02-11T06:25:41.100781484Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZONw9WusWv7gp_9F5F9tQmt","ver":1},"status":{"status":"write"}}},{"id":"1_1613024741.107526_286.1","section":"bucket","name":"xscaju-33","timestamp":"2021-02-11T06:25:41.107526957Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZONw9WusWv7gp_9F5F9tQmt","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024741.135594_287.1","section":"bucket.instance","name":"xscaju-34:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.34","timestamp":"2021-02-11T06:25:41.135594639Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_P2tidfUDvMkQyIFsEqdFcTf","ver":1},"status":{"status":"write"}}},{"id":"1_1613024741.141179_288.1","section":"bucket.instance","name":"xscaju-34:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.34","timestamp":"2021-02-11T06:25:41.141179587Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_P2tidfUDvMkQyIFsEqdFcTf","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024741.142984_289.1","section":"bucket","name":"xscaju-34","timestamp":"2021-02-11T06:25:41.142984206Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SMUniEEMfihR-VQY03kzJxF","ver":1},"status":{"status":"write"}}},{"id":"1_1613024741.147740_290.1","section":"bucket","name":"xscaju-34","timestamp":"2021-02-11T06:25:41.147740630Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SMUniEEMfihR-VQY03kzJxF","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024741.190155_291.1","section":"bucket.instance","name":"xscaju-35:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.35","timestamp":"2021-02-11T06:25:41.190155260Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bxF-qfdodRL7uvp-Gdc5SpJ","ver":1},"status":{"status":"write"}}},{"id":"1_1613024741.195036_292.1","section":"bucket.instance","name":"xscaju-35:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.35","timestamp":"2021-02-11T06:25:41.195036848Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bxF-qfdodRL7uvp-Gdc5SpJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024741.196874_293.1","section":"bucket","name":"xscaju-35","timestamp":"2021-02-11T06:25:41.196874908Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WKxFwlqzDjam6DeHbO6LJrK","ver":1},"status":{"status":"write"}}},{"id":"1_1613024741.202314_294.1","section":"bucket","name":"xscaju-35","timestamp":"2021-02-11T06:25:41.202314216Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WKxFwlqzDjam6DeHbO6LJrK","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024741.227566_295.1","section":"bucket.instance","name":"xscaju-33:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.33","timestamp":"2021-02-11T06:25:41.227566182Z","data":{"read_version":{"tag":"_Y1XZdFxjTuQdDNeNiv617DK","ver":1},"write_version":{"tag":"_Y1XZdFxjTuQdDNeNiv617DK","ver":2},"status":{"status":"write"}}},{"id":"1_1613024741.242076_296.1","section":"bucket.instance","name":"xscaju-33:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.33","timestamp":"2021-02-11T06:25:41.242076851Z","data":{"read_version":{"tag":"_Y1XZdFxjTuQdDNeNiv617DK","ver":1},"write_version":{"tag":"_Y1XZdFxjTuQdDNeNiv617DK","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024742.295410_297.1","section":"bucket.instance","name":"xscaju-34:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.34","timestamp":"2021-02-11T06:25:42.295410472Z","data":{"read_version":{"tag":"_P2tidfUDvMkQyIFsEqdFcTf","ver":1},"write_version":{"tag":"_P2tidfUDvMkQyIFsEqdFcTf","ver":2},"status":{"status":"write"}}},{"id":"1_1613024742.305909_298.1","section":"bucket.instance","name":"xscaju-34:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.34","timestamp":"2021-02-11T06:25:42.305909460Z","data":{"read_version":{"tag":"_P2tidfUDvMkQyIFsEqdFcTf","ver":1},"write_version":{"tag":"_P2tidfUDvMkQyIFsEqdFcTf","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024743.363794_299.1","section":"bucket.instance","name":"xscaju-35:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.35","timestamp":"2021-02-11T06:25:43.363794995Z","data":{"read_version":{"tag":"_bxF-qfdodRL7uvp-Gdc5SpJ","ver":1},"write_version":{"tag":"_bxF-qfdodRL7uvp-Gdc5SpJ","ver":2},"status":{"status":"write"}}},{"id":"1_1613024743.372995_300.1","section":"bucket.instance","name":"xscaju-35:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.35","timestamp":"2021-02-11T06:25:43.372995704Z","data":{"read_version":{"tag":"_bxF-qfdodRL7uvp-Gdc5SpJ","ver":1},"write_version":{"tag":"_bxF-qfdodRL7uvp-Gdc5SpJ","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024743.446380_301.1","section":"bucket.instance","name":"xscaju-36:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.36","timestamp":"2021-02-11T06:25:43.446380603Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SKpaH3MeA1M2VhJGApg7lav","ver":1},"status":{"status":"write"}}},{"id":"1_1613024743.453017_302.1","section":"bucket.instance","name":"xscaju-36:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.36","timestamp":"2021-02-11T06:25:43.453017160Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SKpaH3MeA1M2VhJGApg7lav","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024743.455469_303.1","section":"bucket","name":"xscaju-36","timestamp":"2021-02-11T06:25:43.455469833Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_aFSmLTfOzrHU6jetaDrOjp8","ver":1},"status":{"status":"write"}}},{"id":"1_1613024743.461930_304.1","section":"bucket","name":"xscaju-36","timestamp":"2021-02-11T06:25:43.461930643Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_aFSmLTfOzrHU6jetaDrOjp8","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024743.492333_305.1","section":"bucket.instance","name":"xscaju-37:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.37","timestamp":"2021-02-11T06:25:43.492333619Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Cjc8WiHLvzVFVpjKs0fFrVG","ver":1},"status":{"status":"write"}}},{"id":"1_1613024743.497089_306.1","section":"bucket.instance","name":"xscaju-37:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.37","timestamp":"2021-02-11T06:25:43.497089816Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Cjc8WiHLvzVFVpjKs0fFrVG","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024743.498858_307.1","section":"bucket","name":"xscaju-37","timestamp":"2021-02-11T06:25:43.498858957Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7WecVdsGP5YFX1x8ViLiq1e","ver":1},"status":{"status":"write"}}},{"id":"1_1613024743.503451_308.1","section":"bucket","name":"xscaju-37","timestamp":"2021-02-11T06:25:43.503451797Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7WecVdsGP5YFX1x8ViLiq1e","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024743.542938_309.1","section":"bucket.instance","name":"xscaju-38:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.38","timestamp":"2021-02-11T06:25:43.542938880Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-tTyqozveTUc_iwYr0G-RKw","ver":1},"status":{"status":"write"}}},{"id":"1_1613024743.549571_310.1","section":"bucket.instance","name":"xscaju-38:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.38","timestamp":"2021-02-11T06:25:43.549571216Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-tTyqozveTUc_iwYr0G-RKw","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024743.551903_311.1","section":"bucket","name":"xscaju-38","timestamp":"2021-02-11T06:25:43.551903955Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_NAJwxUoUijTAXWSISc0fRgu","ver":1},"status":{"status":"write"}}},{"id":"1_1613024743.556753_312.1","section":"bucket","name":"xscaju-38","timestamp":"2021-02-11T06:25:43.556753430Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_NAJwxUoUijTAXWSISc0fRgu","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024750.350406_317.1","section":"bucket.instance","name":"xscaju-39:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.39","timestamp":"2021-02-11T06:25:50.350406396Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WCyVAjFo5P4jDOve9UWJjyf","ver":1},"status":{"status":"write"}}},{"id":"1_1613024750.358793_318.1","section":"bucket.instance","name":"xscaju-39:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.39","timestamp":"2021-02-11T06:25:50.358793089Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WCyVAjFo5P4jDOve9UWJjyf","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024750.363526_319.1","section":"bucket","name":"xscaju-39","timestamp":"2021-02-11T06:25:50.363526805Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RQmPlZMEb_u4b1XtxQnquAQ","ver":1},"status":{"status":"write"}}},{"id":"1_1613024750.374527_320.1","section":"bucket","name":"xscaju-39","timestamp":"2021-02-11T06:25:50.374527230Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RQmPlZMEb_u4b1XtxQnquAQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024750.410352_321.1","section":"bucket.instance","name":"xscaju-40:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.40","timestamp":"2021-02-11T06:25:50.410352927Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rO2kG_GMUEGxQo2TvrXh_ea","ver":1},"status":{"status":"write"}}},{"id":"1_1613024750.416134_322.1","section":"bucket.instance","name":"xscaju-40:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.40","timestamp":"2021-02-11T06:25:50.416134960Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rO2kG_GMUEGxQo2TvrXh_ea","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024750.419388_323.1","section":"bucket","name":"xscaju-40","timestamp":"2021-02-11T06:25:50.419388083Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eTxPRJ-BIJLr1m9XX9dTVfp","ver":1},"status":{"status":"write"}}},{"id":"1_1613024750.432139_324.1","section":"bucket","name":"xscaju-40","timestamp":"2021-02-11T06:25:50.432139634Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eTxPRJ-BIJLr1m9XX9dTVfp","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024750.498625_325.1","section":"bucket.instance","name":"xscaju-41:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.41","timestamp":"2021-02-11T06:25:50.498625335Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4HV51C7OPMbwPE8MWSC6HIG","ver":1},"status":{"status":"write"}}},{"id":"1_1613024750.504135_326.1","section":"bucket.instance","name":"xscaju-41:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.41","timestamp":"2021-02-11T06:25:50.504135749Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4HV51C7OPMbwPE8MWSC6HIG","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024750.505803_327.1","section":"bucket","name":"xscaju-41","timestamp":"2021-02-11T06:25:50.505803308Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FtRMvGzj-e8dRznzOVTxQzi","ver":1},"status":{"status":"write"}}},{"id":"1_1613024750.509787_328.1","section":"bucket","name":"xscaju-41","timestamp":"2021-02-11T06:25:50.509787136Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FtRMvGzj-e8dRznzOVTxQzi","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024262.067228_26.1","section":"bucket.instance","name":"xscaju-1:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.1","timestamp":"2021-02-11T06:17:42.067228038Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RvIoHza3ZLDfg2hRKd4PKsh","ver":1},"status":{"status":"write"}}},{"id":"1_1613024262.082142_30.1","section":"bucket.instance","name":"xscaju-1:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.1","timestamp":"2021-02-11T06:17:42.082142380Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RvIoHza3ZLDfg2hRKd4PKsh","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024262.084220_31.1","section":"bucket","name":"xscaju-1","timestamp":"2021-02-11T06:17:42.084220175Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TpoivCStluhaC0Hx2vRcWbz","ver":1},"status":{"status":"write"}}},{"id":"1_1613024262.088459_32.1","section":"bucket","name":"xscaju-1","timestamp":"2021-02-11T06:17:42.088459272Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TpoivCStluhaC0Hx2vRcWbz","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024262.113934_33.1","section":"bucket.instance","name":"xscaju-2:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.2","timestamp":"2021-02-11T06:17:42.113934461Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__hd20jVKBEurKa5-YHlmHCl","ver":1},"status":{"status":"write"}}},{"id":"1_1613024262.120087_34.1","section":"bucket.instance","name":"xscaju-2:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.2","timestamp":"2021-02-11T06:17:42.120087261Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__hd20jVKBEurKa5-YHlmHCl","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024262.122293_35.1","section":"bucket","name":"xscaju-2","timestamp":"2021-02-11T06:17:42.122293620Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XnfQzd3oXvAs9-JK2tnNiPr","ver":1},"status":{"status":"write"}}},{"id":"1_1613024262.127402_36.1","section":"bucket","name":"xscaju-2","timestamp":"2021-02-11T06:17:42.127402916Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XnfQzd3oXvAs9-JK2tnNiPr","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024265.332234_37.1","section":"bucket.instance","name":"xscaju-3:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.3","timestamp":"2021-02-11T06:17:45.332234271Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AWV7DltlSln-5qmsLkFyfua","ver":1},"status":{"status":"write"}}},{"id":"1_1613024265.350497_38.1","section":"bucket.instance","name":"xscaju-3:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.3","timestamp":"2021-02-11T06:17:45.350497485Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AWV7DltlSln-5qmsLkFyfua","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024265.352443_39.1","section":"bucket","name":"xscaju-3","timestamp":"2021-02-11T06:17:45.352443766Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wQHhrUn5k7F9mLuXRB8f1Xz","ver":1},"status":{"status":"write"}}},{"id":"1_1613024265.357008_40.1","section":"bucket","name":"xscaju-3","timestamp":"2021-02-11T06:17:45.357008799Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wQHhrUn5k7F9mLuXRB8f1Xz","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024279.487119_41.1","section":"bucket.instance","name":"xscaju-4:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.4","timestamp":"2021-02-11T06:17:59.487119032Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yaLqVGz15t2aAHd-7NPFL62","ver":1},"status":{"status":"write"}}},{"id":"1_1613024279.492264_43.1","section":"bucket.instance","name":"xscaju-4:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.4","timestamp":"2021-02-11T06:17:59.492264454Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yaLqVGz15t2aAHd-7NPFL62","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024279.494230_44.1","section":"bucket","name":"xscaju-4","timestamp":"2021-02-11T06:17:59.494230677Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_339NBHqlbS70ofKCpOAhJbQ","ver":1},"status":{"status":"write"}}},{"id":"1_1613024279.500216_45.1","section":"bucket","name":"xscaju-4","timestamp":"2021-02-11T06:17:59.500216604Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_339NBHqlbS70ofKCpOAhJbQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024279.522608_46.1","section":"bucket.instance","name":"xscaju-5:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.5","timestamp":"2021-02-11T06:17:59.522608088Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vkQG_Xp99-yJp-ToKtT8kg3","ver":1},"status":{"status":"write"}}},{"id":"1_1613024279.531162_47.1","section":"bucket.instance","name":"xscaju-5:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.5","timestamp":"2021-02-11T06:17:59.531162028Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vkQG_Xp99-yJp-ToKtT8kg3","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024279.533140_48.1","section":"bucket","name":"xscaju-5","timestamp":"2021-02-11T06:17:59.533140603Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SVeKSQ9gdskN7i7U-nYLEgi","ver":1},"status":{"status":"write"}}},{"id":"1_1613024279.540314_49.1","section":"bucket","name":"xscaju-5","timestamp":"2021-02-11T06:17:59.540314431Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SVeKSQ9gdskN7i7U-nYLEgi","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024279.587364_50.1","section":"bucket.instance","name":"xscaju-6:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.6","timestamp":"2021-02-11T06:17:59.587364825Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rte0LgEFVcWvYNPAx9nNs_n","ver":1},"status":{"status":"write"}}},{"id":"1_1613024279.595650_51.1","section":"bucket.instance","name":"xscaju-6:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.6","timestamp":"2021-02-11T06:17:59.595650557Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rte0LgEFVcWvYNPAx9nNs_n","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024279.599110_52.1","section":"bucket","name":"xscaju-6","timestamp":"2021-02-11T06:17:59.599110042Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9FvsuLFeId2rWrC78ZduVOd","ver":1},"status":{"status":"write"}}},{"id":"1_1613024279.610246_53.1","section":"bucket","name":"xscaju-6","timestamp":"2021-02-11T06:17:59.610246072Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9FvsuLFeId2rWrC78ZduVOd","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024294.216703_54.1","section":"bucket.instance","name":"xscaju-7:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.7","timestamp":"2021-02-11T06:18:14.216703877Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jWdNA4N3P_5daffu-K1YRGJ","ver":1},"status":{"status":"write"}}},{"id":"1_1613024294.222678_57.1","section":"bucket.instance","name":"xscaju-7:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.7","timestamp":"2021-02-11T06:18:14.222678059Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jWdNA4N3P_5daffu-K1YRGJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024294.224967_58.1","section":"bucket","name":"xscaju-7","timestamp":"2021-02-11T06:18:14.224967555Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_E45sVYzIU0DaBEmS8x7_Rdy","ver":1},"status":{"status":"write"}}},{"id":"1_1613024294.230021_59.1","section":"bucket","name":"xscaju-7","timestamp":"2021-02-11T06:18:14.230021669Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_E45sVYzIU0DaBEmS8x7_Rdy","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024294.249488_60.1","section":"bucket.instance","name":"xscaju-8:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.8","timestamp":"2021-02-11T06:18:14.249488401Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kfebcc_EHBe3uMLbBs7dDoL","ver":1},"status":{"status":"write"}}},{"id":"1_1613024294.255631_61.1","section":"bucket.instance","name":"xscaju-8:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.8","timestamp":"2021-02-11T06:18:14.255631956Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kfebcc_EHBe3uMLbBs7dDoL","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024294.257842_62.1","section":"bucket","name":"xscaju-8","timestamp":"2021-02-11T06:18:14.257842711Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Wh2l0tE943p3oAEfB-UgE1m","ver":1},"status":{"status":"write"}}},{"id":"1_1613024294.263232_63.1","section":"bucket","name":"xscaju-8","timestamp":"2021-02-11T06:18:14.263232235Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Wh2l0tE943p3oAEfB-UgE1m","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024294.307011_64.1","section":"bucket.instance","name":"xscaju-9:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.9","timestamp":"2021-02-11T06:18:14.307011641Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5js7vqwL9Cu1tRXnOerclkl","ver":1},"status":{"status":"write"}}},{"id":"1_1613024294.313905_65.1","section":"bucket.instance","name":"xscaju-9:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.9","timestamp":"2021-02-11T06:18:14.313905047Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5js7vqwL9Cu1tRXnOerclkl","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024294.315946_66.1","section":"bucket","name":"xscaju-9","timestamp":"2021-02-11T06:18:14.315946160Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UWAgQgKN2X4AJl_kFPBitiQ","ver":1},"status":{"status":"write"}}},{"id":"1_1613024294.322067_67.1","section":"bucket","name":"xscaju-9","timestamp":"2021-02-11T06:18:14.322067563Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UWAgQgKN2X4AJl_kFPBitiQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024300.918033_68.1","section":"bucket","name":"xscaju-7","timestamp":"2021-02-11T06:18:20.918033480Z","data":{"read_version":{"tag":"_E45sVYzIU0DaBEmS8x7_Rdy","ver":1},"write_version":{"tag":"_E45sVYzIU0DaBEmS8x7_Rdy","ver":2},"status":{"status":"remove"}}},{"id":"1_1613024300.924725_70.1","section":"bucket","name":"xscaju-7","timestamp":"2021-02-11T06:18:20.924725131Z","data":{"read_version":{"tag":"_E45sVYzIU0DaBEmS8x7_Rdy","ver":1},"write_version":{"tag":"_E45sVYzIU0DaBEmS8x7_Rdy","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024300.998791_71.1","section":"bucket","name":"xscaju-8","timestamp":"2021-02-11T06:18:20.998791991Z","data":{"read_version":{"tag":"_Wh2l0tE943p3oAEfB-UgE1m","ver":1},"write_version":{"tag":"_Wh2l0tE943p3oAEfB-UgE1m","ver":2},"status":{"status":"remove"}}},{"id":"1_1613024301.006963_72.1","section":"bucket","name":"xscaju-8","timestamp":"2021-02-11T06:18:21.006963296Z","data":{"read_version":{"tag":"_Wh2l0tE943p3oAEfB-UgE1m","ver":1},"write_version":{"tag":"_Wh2l0tE943p3oAEfB-UgE1m","ver":2},"status":{"status":"complete"}}},{"id":"1_1613024301.135430_73.1","section":"bucket","name":"xscaju-9","timestamp":"2021-02-11T06:18:21.135430186Z","data":{"read_version":{"tag":"_UWAgQgKN2X4AJl_kFPBitiQ","ver":1},"write_version":{"tag":"_UWAgQgKN2X4AJl_kFPBitiQ","ver":2},"status":{"status":"remove"}}},{"id":"1_1613024301.142235_74.1","section":"bucket","name":"xscaju-9","timestamp":"2021-02-11T06:18:21.142235547Z","data":{"read_version":{"tag":"_UWAgQgKN2X4AJl_kFPBitiQ","ver":1},"write_version":{"tag":"_UWAgQgKN2X4AJl_kFPBitiQ","ver":2},"status":{"status":"complete"}}}] 2021-02-11T06:28:38.815 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '2afe6ddd-1cf8-433a-ae51-67de25931f7b', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:28:38.816 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '2afe6ddd-1cf8-433a-ae51-67de25931f7b', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:28:38.816 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 2afe6ddd-1cf8-433a-ae51-67de25931f7b --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:28:38.854 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:28:38.855 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:28:39.130 INFO:teuthology.orchestra.run.smithi061.stdout:[{"id":"1_1613024818.338835_259.1","section":"bucket.instance","name":"xscaju-42:e4b3955d-973b-4dd2-ae54-b7e681fbeb21.5034.1","timestamp":"2021-02-11T06:26:58.338835032Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9UGbuvkoI0Ws7knG_XbXE5-","ver":1},"status":{"status":"write"}}},{"id":"1_1613024818.344159_265.1","section":"bucket.instance","name":"xscaju-42:e4b3955d-973b-4dd2-ae54-b7e681fbeb21.5034.1","timestamp":"2021-02-11T06:26:58.344159628Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9UGbuvkoI0Ws7knG_XbXE5-","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024818.345967_266.1","section":"bucket","name":"xscaju-42","timestamp":"2021-02-11T06:26:58.345967501Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BibNrDCQfTM1NO7W0BfvNyN","ver":1},"status":{"status":"write"}}},{"id":"1_1613024818.351571_267.1","section":"bucket","name":"xscaju-42","timestamp":"2021-02-11T06:26:58.351571391Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BibNrDCQfTM1NO7W0BfvNyN","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024818.400993_268.1","section":"bucket.instance","name":"xscaju-43:e4b3955d-973b-4dd2-ae54-b7e681fbeb21.5034.2","timestamp":"2021-02-11T06:26:58.400993108Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8FOSAH5DuoSD_L_kGn6zJ-R","ver":1},"status":{"status":"write"}}},{"id":"1_1613024818.409009_269.1","section":"bucket.instance","name":"xscaju-43:e4b3955d-973b-4dd2-ae54-b7e681fbeb21.5034.2","timestamp":"2021-02-11T06:26:58.409009093Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8FOSAH5DuoSD_L_kGn6zJ-R","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024818.411049_270.1","section":"bucket","name":"xscaju-43","timestamp":"2021-02-11T06:26:58.411049916Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zG9Hf9fR595KlguMsPpTkhS","ver":1},"status":{"status":"write"}}},{"id":"1_1613024818.417656_271.1","section":"bucket","name":"xscaju-43","timestamp":"2021-02-11T06:26:58.417656173Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zG9Hf9fR595KlguMsPpTkhS","ver":1},"status":{"status":"complete"}}}] 2021-02-11T06:28:39.131 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '2afe6ddd-1cf8-433a-ae51-67de25931f7b', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:28:39.131 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '2afe6ddd-1cf8-433a-ae51-67de25931f7b', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:28:39.131 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 2afe6ddd-1cf8-433a-ae51-67de25931f7b --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:28:39.172 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:28:39.172 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:28:39.415 INFO:teuthology.orchestra.run.smithi172.stdout:[{"id":"1_1613024818.338835_259.1","section":"bucket.instance","name":"xscaju-42:e4b3955d-973b-4dd2-ae54-b7e681fbeb21.5034.1","timestamp":"2021-02-11T06:26:58.338835032Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9UGbuvkoI0Ws7knG_XbXE5-","ver":1},"status":{"status":"write"}}},{"id":"1_1613024818.344159_265.1","section":"bucket.instance","name":"xscaju-42:e4b3955d-973b-4dd2-ae54-b7e681fbeb21.5034.1","timestamp":"2021-02-11T06:26:58.344159628Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9UGbuvkoI0Ws7knG_XbXE5-","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024818.345967_266.1","section":"bucket","name":"xscaju-42","timestamp":"2021-02-11T06:26:58.345967501Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BibNrDCQfTM1NO7W0BfvNyN","ver":1},"status":{"status":"write"}}},{"id":"1_1613024818.351571_267.1","section":"bucket","name":"xscaju-42","timestamp":"2021-02-11T06:26:58.351571391Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BibNrDCQfTM1NO7W0BfvNyN","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024818.400993_268.1","section":"bucket.instance","name":"xscaju-43:e4b3955d-973b-4dd2-ae54-b7e681fbeb21.5034.2","timestamp":"2021-02-11T06:26:58.400993108Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8FOSAH5DuoSD_L_kGn6zJ-R","ver":1},"status":{"status":"write"}}},{"id":"1_1613024818.409009_269.1","section":"bucket.instance","name":"xscaju-43:e4b3955d-973b-4dd2-ae54-b7e681fbeb21.5034.2","timestamp":"2021-02-11T06:26:58.409009093Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8FOSAH5DuoSD_L_kGn6zJ-R","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024818.411049_270.1","section":"bucket","name":"xscaju-43","timestamp":"2021-02-11T06:26:58.411049916Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zG9Hf9fR595KlguMsPpTkhS","ver":1},"status":{"status":"write"}}},{"id":"1_1613024818.417656_271.1","section":"bucket","name":"xscaju-43","timestamp":"2021-02-11T06:26:58.417656173Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zG9Hf9fR595KlguMsPpTkhS","ver":1},"status":{"status":"complete"}}}] 2021-02-11T06:28:39.415 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '2afe6ddd-1cf8-433a-ae51-67de25931f7b', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:28:39.416 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '2afe6ddd-1cf8-433a-ae51-67de25931f7b', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:28:39.416 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 2afe6ddd-1cf8-433a-ae51-67de25931f7b --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:28:39.453 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:28:39.453 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:28:39.738 INFO:teuthology.orchestra.run.smithi061.stdout:[{"id":"1_1613024818.338835_259.1","section":"bucket.instance","name":"xscaju-42:e4b3955d-973b-4dd2-ae54-b7e681fbeb21.5034.1","timestamp":"2021-02-11T06:26:58.338835032Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9UGbuvkoI0Ws7knG_XbXE5-","ver":1},"status":{"status":"write"}}},{"id":"1_1613024818.344159_265.1","section":"bucket.instance","name":"xscaju-42:e4b3955d-973b-4dd2-ae54-b7e681fbeb21.5034.1","timestamp":"2021-02-11T06:26:58.344159628Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9UGbuvkoI0Ws7knG_XbXE5-","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024818.345967_266.1","section":"bucket","name":"xscaju-42","timestamp":"2021-02-11T06:26:58.345967501Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BibNrDCQfTM1NO7W0BfvNyN","ver":1},"status":{"status":"write"}}},{"id":"1_1613024818.351571_267.1","section":"bucket","name":"xscaju-42","timestamp":"2021-02-11T06:26:58.351571391Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BibNrDCQfTM1NO7W0BfvNyN","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024818.400993_268.1","section":"bucket.instance","name":"xscaju-43:e4b3955d-973b-4dd2-ae54-b7e681fbeb21.5034.2","timestamp":"2021-02-11T06:26:58.400993108Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8FOSAH5DuoSD_L_kGn6zJ-R","ver":1},"status":{"status":"write"}}},{"id":"1_1613024818.409009_269.1","section":"bucket.instance","name":"xscaju-43:e4b3955d-973b-4dd2-ae54-b7e681fbeb21.5034.2","timestamp":"2021-02-11T06:26:58.409009093Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8FOSAH5DuoSD_L_kGn6zJ-R","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024818.411049_270.1","section":"bucket","name":"xscaju-43","timestamp":"2021-02-11T06:26:58.411049916Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zG9Hf9fR595KlguMsPpTkhS","ver":1},"status":{"status":"write"}}},{"id":"1_1613024818.417656_271.1","section":"bucket","name":"xscaju-43","timestamp":"2021-02-11T06:26:58.417656173Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zG9Hf9fR595KlguMsPpTkhS","ver":1},"status":{"status":"complete"}}}] 2021-02-11T06:28:39.739 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '2afe6ddd-1cf8-433a-ae51-67de25931f7b', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:28:39.739 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '2afe6ddd-1cf8-433a-ae51-67de25931f7b', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:28:39.740 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 2afe6ddd-1cf8-433a-ae51-67de25931f7b --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:28:39.777 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:28:39.778 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:28:40.034 INFO:teuthology.orchestra.run.smithi172.stdout:[{"id":"1_1613024818.338835_259.1","section":"bucket.instance","name":"xscaju-42:e4b3955d-973b-4dd2-ae54-b7e681fbeb21.5034.1","timestamp":"2021-02-11T06:26:58.338835032Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9UGbuvkoI0Ws7knG_XbXE5-","ver":1},"status":{"status":"write"}}},{"id":"1_1613024818.344159_265.1","section":"bucket.instance","name":"xscaju-42:e4b3955d-973b-4dd2-ae54-b7e681fbeb21.5034.1","timestamp":"2021-02-11T06:26:58.344159628Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9UGbuvkoI0Ws7knG_XbXE5-","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024818.345967_266.1","section":"bucket","name":"xscaju-42","timestamp":"2021-02-11T06:26:58.345967501Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BibNrDCQfTM1NO7W0BfvNyN","ver":1},"status":{"status":"write"}}},{"id":"1_1613024818.351571_267.1","section":"bucket","name":"xscaju-42","timestamp":"2021-02-11T06:26:58.351571391Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BibNrDCQfTM1NO7W0BfvNyN","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024818.400993_268.1","section":"bucket.instance","name":"xscaju-43:e4b3955d-973b-4dd2-ae54-b7e681fbeb21.5034.2","timestamp":"2021-02-11T06:26:58.400993108Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8FOSAH5DuoSD_L_kGn6zJ-R","ver":1},"status":{"status":"write"}}},{"id":"1_1613024818.409009_269.1","section":"bucket.instance","name":"xscaju-43:e4b3955d-973b-4dd2-ae54-b7e681fbeb21.5034.2","timestamp":"2021-02-11T06:26:58.409009093Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8FOSAH5DuoSD_L_kGn6zJ-R","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024818.411049_270.1","section":"bucket","name":"xscaju-43","timestamp":"2021-02-11T06:26:58.411049916Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zG9Hf9fR595KlguMsPpTkhS","ver":1},"status":{"status":"write"}}},{"id":"1_1613024818.417656_271.1","section":"bucket","name":"xscaju-43","timestamp":"2021-02-11T06:26:58.417656173Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zG9Hf9fR595KlguMsPpTkhS","ver":1},"status":{"status":"complete"}}}] 2021-02-11T06:28:40.035 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '0d061ff0-ede8-4baa-beef-d975d317d505', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:28:40.035 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '0d061ff0-ede8-4baa-beef-d975d317d505', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:28:40.035 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 0d061ff0-ede8-4baa-beef-d975d317d505 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:28:40.073 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:28:40.073 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:28:40.350 INFO:teuthology.orchestra.run.smithi061.stdout:[{"id":"1_1613024879.587590_88.1","section":"bucket.instance","name":"xscaju-44:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5174.1","timestamp":"2021-02-11T06:27:59.587590333Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BjCp0HZGY76_u_AFXYS7gZd","ver":1},"status":{"status":"write"}}},{"id":"1_1613024879.594259_89.1","section":"bucket.instance","name":"xscaju-44:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5174.1","timestamp":"2021-02-11T06:27:59.594259264Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BjCp0HZGY76_u_AFXYS7gZd","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024879.596286_90.1","section":"bucket","name":"xscaju-44","timestamp":"2021-02-11T06:27:59.596286282Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7KuybERlCz-8KLUumVHDuz0","ver":1},"status":{"status":"write"}}},{"id":"1_1613024879.601775_91.1","section":"bucket","name":"xscaju-44","timestamp":"2021-02-11T06:27:59.601775577Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7KuybERlCz-8KLUumVHDuz0","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024879.647219_92.1","section":"bucket.instance","name":"xscaju-45:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5174.2","timestamp":"2021-02-11T06:27:59.647219009Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JgtEny2gt0YQBOSzHWGzORi","ver":1},"status":{"status":"write"}}},{"id":"1_1613024879.653023_93.1","section":"bucket.instance","name":"xscaju-45:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5174.2","timestamp":"2021-02-11T06:27:59.653023075Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JgtEny2gt0YQBOSzHWGzORi","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024879.655036_94.1","section":"bucket","name":"xscaju-45","timestamp":"2021-02-11T06:27:59.655036437Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ok_7i6IrSZEblNM1nUw92_J","ver":1},"status":{"status":"write"}}},{"id":"1_1613024879.661926_95.1","section":"bucket","name":"xscaju-45","timestamp":"2021-02-11T06:27:59.661926439Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ok_7i6IrSZEblNM1nUw92_J","ver":1},"status":{"status":"complete"}}}] 2021-02-11T06:28:40.351 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '0d061ff0-ede8-4baa-beef-d975d317d505', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:28:40.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '0d061ff0-ede8-4baa-beef-d975d317d505', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:28:40.351 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 0d061ff0-ede8-4baa-beef-d975d317d505 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:28:40.390 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:28:40.390 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:28:40.646 INFO:teuthology.orchestra.run.smithi172.stdout:[{"id":"1_1613024879.587590_88.1","section":"bucket.instance","name":"xscaju-44:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5174.1","timestamp":"2021-02-11T06:27:59.587590333Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BjCp0HZGY76_u_AFXYS7gZd","ver":1},"status":{"status":"write"}}},{"id":"1_1613024879.594259_89.1","section":"bucket.instance","name":"xscaju-44:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5174.1","timestamp":"2021-02-11T06:27:59.594259264Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BjCp0HZGY76_u_AFXYS7gZd","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024879.596286_90.1","section":"bucket","name":"xscaju-44","timestamp":"2021-02-11T06:27:59.596286282Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7KuybERlCz-8KLUumVHDuz0","ver":1},"status":{"status":"write"}}},{"id":"1_1613024879.601775_91.1","section":"bucket","name":"xscaju-44","timestamp":"2021-02-11T06:27:59.601775577Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7KuybERlCz-8KLUumVHDuz0","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024879.647219_92.1","section":"bucket.instance","name":"xscaju-45:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5174.2","timestamp":"2021-02-11T06:27:59.647219009Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JgtEny2gt0YQBOSzHWGzORi","ver":1},"status":{"status":"write"}}},{"id":"1_1613024879.653023_93.1","section":"bucket.instance","name":"xscaju-45:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5174.2","timestamp":"2021-02-11T06:27:59.653023075Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JgtEny2gt0YQBOSzHWGzORi","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024879.655036_94.1","section":"bucket","name":"xscaju-45","timestamp":"2021-02-11T06:27:59.655036437Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ok_7i6IrSZEblNM1nUw92_J","ver":1},"status":{"status":"write"}}},{"id":"1_1613024879.661926_95.1","section":"bucket","name":"xscaju-45","timestamp":"2021-02-11T06:27:59.661926439Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ok_7i6IrSZEblNM1nUw92_J","ver":1},"status":{"status":"complete"}}}] 2021-02-11T06:28:40.646 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '0d061ff0-ede8-4baa-beef-d975d317d505', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:28:40.646 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '0d061ff0-ede8-4baa-beef-d975d317d505', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:28:40.646 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 0d061ff0-ede8-4baa-beef-d975d317d505 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:28:40.685 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:28:40.685 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:28:40.963 INFO:teuthology.orchestra.run.smithi061.stdout:[{"id":"1_1613024879.587590_88.1","section":"bucket.instance","name":"xscaju-44:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5174.1","timestamp":"2021-02-11T06:27:59.587590333Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BjCp0HZGY76_u_AFXYS7gZd","ver":1},"status":{"status":"write"}}},{"id":"1_1613024879.594259_89.1","section":"bucket.instance","name":"xscaju-44:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5174.1","timestamp":"2021-02-11T06:27:59.594259264Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BjCp0HZGY76_u_AFXYS7gZd","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024879.596286_90.1","section":"bucket","name":"xscaju-44","timestamp":"2021-02-11T06:27:59.596286282Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7KuybERlCz-8KLUumVHDuz0","ver":1},"status":{"status":"write"}}},{"id":"1_1613024879.601775_91.1","section":"bucket","name":"xscaju-44","timestamp":"2021-02-11T06:27:59.601775577Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7KuybERlCz-8KLUumVHDuz0","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024879.647219_92.1","section":"bucket.instance","name":"xscaju-45:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5174.2","timestamp":"2021-02-11T06:27:59.647219009Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JgtEny2gt0YQBOSzHWGzORi","ver":1},"status":{"status":"write"}}},{"id":"1_1613024879.653023_93.1","section":"bucket.instance","name":"xscaju-45:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5174.2","timestamp":"2021-02-11T06:27:59.653023075Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JgtEny2gt0YQBOSzHWGzORi","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024879.655036_94.1","section":"bucket","name":"xscaju-45","timestamp":"2021-02-11T06:27:59.655036437Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ok_7i6IrSZEblNM1nUw92_J","ver":1},"status":{"status":"write"}}},{"id":"1_1613024879.661926_95.1","section":"bucket","name":"xscaju-45","timestamp":"2021-02-11T06:27:59.661926439Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ok_7i6IrSZEblNM1nUw92_J","ver":1},"status":{"status":"complete"}}}] 2021-02-11T06:28:40.963 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '0d061ff0-ede8-4baa-beef-d975d317d505', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:28:40.963 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '0d061ff0-ede8-4baa-beef-d975d317d505', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:28:40.964 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 0d061ff0-ede8-4baa-beef-d975d317d505 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:28:41.003 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:28:41.003 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:28:41.256 INFO:teuthology.orchestra.run.smithi172.stdout:[{"id":"1_1613024879.587590_88.1","section":"bucket.instance","name":"xscaju-44:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5174.1","timestamp":"2021-02-11T06:27:59.587590333Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BjCp0HZGY76_u_AFXYS7gZd","ver":1},"status":{"status":"write"}}},{"id":"1_1613024879.594259_89.1","section":"bucket.instance","name":"xscaju-44:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5174.1","timestamp":"2021-02-11T06:27:59.594259264Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BjCp0HZGY76_u_AFXYS7gZd","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024879.596286_90.1","section":"bucket","name":"xscaju-44","timestamp":"2021-02-11T06:27:59.596286282Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7KuybERlCz-8KLUumVHDuz0","ver":1},"status":{"status":"write"}}},{"id":"1_1613024879.601775_91.1","section":"bucket","name":"xscaju-44","timestamp":"2021-02-11T06:27:59.601775577Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7KuybERlCz-8KLUumVHDuz0","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024879.647219_92.1","section":"bucket.instance","name":"xscaju-45:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5174.2","timestamp":"2021-02-11T06:27:59.647219009Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JgtEny2gt0YQBOSzHWGzORi","ver":1},"status":{"status":"write"}}},{"id":"1_1613024879.653023_93.1","section":"bucket.instance","name":"xscaju-45:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5174.2","timestamp":"2021-02-11T06:27:59.653023075Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JgtEny2gt0YQBOSzHWGzORi","ver":1},"status":{"status":"complete"}}},{"id":"1_1613024879.655036_94.1","section":"bucket","name":"xscaju-45","timestamp":"2021-02-11T06:27:59.655036437Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ok_7i6IrSZEblNM1nUw92_J","ver":1},"status":{"status":"write"}}},{"id":"1_1613024879.661926_95.1","section":"bucket","name":"xscaju-45","timestamp":"2021-02-11T06:27:59.661926439Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ok_7i6IrSZEblNM1nUw92_J","ver":1},"status":{"status":"complete"}}}] 2021-02-11T06:28:41.256 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:28:41.256 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:28:41.257 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T06:28:41.294 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:28:41.294 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:28:41.619 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:28:41.619 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:28:41.619 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T06:28:41.656 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:28:41.657 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:28:42.102 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:28:42.102 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:28:42.103 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T06:28:42.139 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:28:42.139 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:28:42.420 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:28:53.648 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:28:53.649 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T06:28:53.687 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:28:53.688 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:28:54.092 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '7c4a812f-81e4-4a72-acce-0ee18589648c', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:28:54.092 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '7c4a812f-81e4-4a72-acce-0ee18589648c', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:28:54.092 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 7c4a812f-81e4-4a72-acce-0ee18589648c --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:28:54.132 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:28:54.132 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:28:54.387 INFO:teuthology.orchestra.run.smithi061.stdout:[] 2021-02-11T06:29:13.701 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '7c4a812f-81e4-4a72-acce-0ee18589648c', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:29:13.701 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '7c4a812f-81e4-4a72-acce-0ee18589648c', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:29:13.702 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 7c4a812f-81e4-4a72-acce-0ee18589648c --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:29:13.741 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:29:13.741 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:29:13.991 INFO:teuthology.orchestra.run.smithi172.stdout:[] 2021-02-11T06:29:13.991 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '7c4a812f-81e4-4a72-acce-0ee18589648c', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:29:13.991 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '7c4a812f-81e4-4a72-acce-0ee18589648c', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:29:13.991 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 7c4a812f-81e4-4a72-acce-0ee18589648c --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:29:14.030 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:29:14.031 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:29:14.313 INFO:teuthology.orchestra.run.smithi061.stdout:[] 2021-02-11T06:29:14.313 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '7c4a812f-81e4-4a72-acce-0ee18589648c', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:29:14.314 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '7c4a812f-81e4-4a72-acce-0ee18589648c', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:29:14.314 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 7c4a812f-81e4-4a72-acce-0ee18589648c --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:29:14.351 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:29:14.351 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:29:14.612 INFO:teuthology.orchestra.run.smithi172.stdout:[] 2021-02-11T06:29:14.612 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '2afe6ddd-1cf8-433a-ae51-67de25931f7b', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:29:14.612 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '2afe6ddd-1cf8-433a-ae51-67de25931f7b', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:29:14.613 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 2afe6ddd-1cf8-433a-ae51-67de25931f7b --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:29:14.654 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:29:14.654 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:29:14.932 INFO:teuthology.orchestra.run.smithi061.stdout:[] 2021-02-11T06:29:14.933 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '2afe6ddd-1cf8-433a-ae51-67de25931f7b', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:29:14.933 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '2afe6ddd-1cf8-433a-ae51-67de25931f7b', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:29:14.933 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 2afe6ddd-1cf8-433a-ae51-67de25931f7b --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:29:14.972 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:29:14.972 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:29:15.218 INFO:teuthology.orchestra.run.smithi172.stdout:[] 2021-02-11T06:29:15.218 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '2afe6ddd-1cf8-433a-ae51-67de25931f7b', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:29:15.219 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '2afe6ddd-1cf8-433a-ae51-67de25931f7b', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:29:15.219 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 2afe6ddd-1cf8-433a-ae51-67de25931f7b --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:29:15.257 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:29:15.257 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:29:15.524 INFO:teuthology.orchestra.run.smithi061.stdout:[] 2021-02-11T06:29:15.524 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '2afe6ddd-1cf8-433a-ae51-67de25931f7b', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:29:15.524 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '2afe6ddd-1cf8-433a-ae51-67de25931f7b', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:29:15.525 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 2afe6ddd-1cf8-433a-ae51-67de25931f7b --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:29:15.564 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:29:16.019 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:29:16.020 INFO:teuthology.orchestra.run.smithi172.stdout:[] 2021-02-11T06:29:16.020 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '0d061ff0-ede8-4baa-beef-d975d317d505', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:29:16.021 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '0d061ff0-ede8-4baa-beef-d975d317d505', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:29:16.021 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 0d061ff0-ede8-4baa-beef-d975d317d505 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:29:16.062 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:29:16.062 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:29:16.330 INFO:teuthology.orchestra.run.smithi061.stdout:[] 2021-02-11T06:29:16.331 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '0d061ff0-ede8-4baa-beef-d975d317d505', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:29:16.331 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '0d061ff0-ede8-4baa-beef-d975d317d505', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:29:16.331 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 0d061ff0-ede8-4baa-beef-d975d317d505 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:29:16.369 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:29:16.369 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:29:16.642 INFO:teuthology.orchestra.run.smithi172.stdout:[] 2021-02-11T06:29:16.642 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '0d061ff0-ede8-4baa-beef-d975d317d505', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:29:16.642 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '0d061ff0-ede8-4baa-beef-d975d317d505', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:29:16.642 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 0d061ff0-ede8-4baa-beef-d975d317d505 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:29:16.681 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:29:16.681 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:29:16.963 INFO:teuthology.orchestra.run.smithi061.stdout:[] 2021-02-11T06:29:16.963 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '0d061ff0-ede8-4baa-beef-d975d317d505', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:29:16.964 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '0d061ff0-ede8-4baa-beef-d975d317d505', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:29:16.964 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 0d061ff0-ede8-4baa-beef-d975d317d505 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:29:17.003 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:29:17.003 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:29:17.255 INFO:teuthology.orchestra.run.smithi172.stdout:[] 2021-02-11T06:29:17.255 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_multi_period_incremental_sync ... ok 2021-02-11T06:29:17.256 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xscaju-46 2021-02-11T06:29:17.296 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xscaju-47 2021-02-11T06:29:17.374 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xscaju-48 2021-02-11T06:29:17.575 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:29:17.575 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:29:17.575 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:29:17.613 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:29:17.614 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:29:17.826 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=0d061ff0-ede8-4baa-beef-d975d317d505 2021-02-11T06:29:18.215 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613024957.411871_109.1","last_update":"2021-02-11T06:29:17.411871Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:29:18.216 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T06:29:18.216 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:29:18.216 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:29:18.216 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:29:18.257 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:29:18.258 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:29:18.498 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613024957.411871_109.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:29:17.411871Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:29:18.499 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T06:29:18.499 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T06:29:18.499 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:29:18.499 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:29:18.499 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:29:18.541 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:29:18.542 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:29:18.819 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613024957.411871_109.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:29:17.411871Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:29:18.819 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T06:29:18.819 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:29:18.819 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:29:18.819 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:29:18.820 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:29:18.859 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:29:18.859 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:29:19.117 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613024957.411871_109.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:29:17.411871Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:29:19.117 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:29:19.117 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:29:19.117 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:29:19.118 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:29:19.159 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:29:19.159 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:29:19.442 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"00000000000000000000:00000000000000004635","last_update":"2021-02-11T06:21:59.639441Z"},{"marker":"00000000000000000000:00000000000000000780","last_update":"2021-02-11T06:25:50.534339Z"},{"marker":"00000000000000000000:00000000000000002507","last_update":"2021-02-11T06:21:37.634942Z"},{"marker":"00000000000000000000:00000000000000003569","last_update":"2021-02-11T06:29:17.877002Z"}] 2021-02-11T06:29:19.442 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-02-11T06:29:19.442 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:29:19.442 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:29:19.442 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:29:19.485 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:29:19.485 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:29:19.746 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":15153533235123629405},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004635","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:21:59.639441Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000780","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:25:50.534339Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002507","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:21:37.634942Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003569","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:29:17.877002Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T06:29:19.746 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-02-11T06:29:19.746 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:29:19.747 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:29:19.747 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:29:19.790 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:29:19.790 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:29:20.058 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"00000000000000000000:00000000000000004635","last_update":"2021-02-11T06:21:59.639441Z"},{"marker":"00000000000000000000:00000000000000000780","last_update":"2021-02-11T06:25:50.534339Z"},{"marker":"00000000000000000000:00000000000000002507","last_update":"2021-02-11T06:21:37.634942Z"},{"marker":"00000000000000000000:00000000000000003569","last_update":"2021-02-11T06:29:17.877002Z"}] 2021-02-11T06:29:20.058 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-02-11T06:29:20.059 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:29:20.059 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:29:20.059 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:29:20.143 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:29:20.143 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:29:20.392 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":15040254564068855740},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004635","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:21:59.639441Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000780","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:25:50.534339Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002507","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:21:37.634942Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003569","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:29:17.877002Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T06:29:20.392 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-02-11T06:29:20.392 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:29:20.392 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:29:20.392 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:29:20.474 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:29:20.474 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:29:20.750 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"00000000000000000000:00000000000000004635","last_update":"2021-02-11T06:21:59.639441Z"},{"marker":"00000000000000000000:00000000000000000780","last_update":"2021-02-11T06:25:50.534339Z"},{"marker":"00000000000000000000:00000000000000002507","last_update":"2021-02-11T06:21:37.634942Z"},{"marker":"00000000000000000000:00000000000000003569","last_update":"2021-02-11T06:29:17.877002Z"}] 2021-02-11T06:29:20.750 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-02-11T06:29:20.751 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:29:20.751 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:29:20.751 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:29:20.793 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:29:20.793 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:29:21.036 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":1899655024930128290},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004635","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:21:59.639441Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000780","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:25:50.534339Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002507","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:21:37.634942Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003569","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:29:17.877002Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T06:29:21.037 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-02-11T06:29:21.037 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:29:21.037 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:29:21.037 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:29:21.115 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:29:21.115 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:29:21.397 INFO:teuthology.orchestra.run.smithi172.stdout:[{"marker":"00000000000000000000:00000000000000004635","last_update":"2021-02-11T06:21:59.791157Z"},{"marker":"00000000000000000000:00000000000000000780","last_update":"2021-02-11T06:25:50.571878Z"},{"marker":"00000000000000000000:00000000000000002374","last_update":"2021-02-11T06:21:28.667313Z"},{"marker":"00000000000000000000:00000000000000003569","last_update":"2021-02-11T06:29:17.830216Z"}] 2021-02-11T06:29:21.397 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-02-11T06:29:21.398 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:29:21.398 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:29:21.398 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:29:21.441 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:29:21.441 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:29:21.704 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":3699056310506273018},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004635","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:21:59.791157Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000780","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:25:50.571878Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002374","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:21:28.667313Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003569","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:29:17.830216Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T06:29:21.704 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-02-11T06:29:21.705 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:29:21.705 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:29:21.705 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:29:21.743 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:29:21.743 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:29:22.006 INFO:teuthology.orchestra.run.smithi172.stdout:[{"marker":"00000000000000000000:00000000000000004635","last_update":"2021-02-11T06:21:59.791157Z"},{"marker":"00000000000000000000:00000000000000000780","last_update":"2021-02-11T06:25:50.571878Z"},{"marker":"00000000000000000000:00000000000000002374","last_update":"2021-02-11T06:21:28.667313Z"},{"marker":"00000000000000000000:00000000000000003569","last_update":"2021-02-11T06:29:17.830216Z"}] 2021-02-11T06:29:22.006 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-02-11T06:29:22.007 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:29:22.007 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:29:22.007 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:29:22.049 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:29:22.049 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:29:22.308 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":995053126825359944},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004635","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:21:59.791157Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000780","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:25:50.571878Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002374","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:21:28.667313Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003569","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:29:17.830216Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T06:29:22.309 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-02-11T06:29:22.309 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:29:22.309 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:29:22.309 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:29:22.392 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:29:22.392 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:29:22.648 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"00000000000000000000:00000000000000004635","last_update":"2021-02-11T06:21:59.817702Z"},{"marker":"00000000000000000000:00000000000000000780","last_update":"2021-02-11T06:25:50.349364Z"},{"marker":"00000000000000000000:00000000000000002374","last_update":"2021-02-11T06:21:28.930530Z"},{"marker":"00000000000000000000:00000000000000003569","last_update":"2021-02-11T06:29:17.796678Z"}] 2021-02-11T06:29:22.648 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-02-11T06:29:22.648 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:29:22.648 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:29:22.648 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:29:22.729 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:29:22.729 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:29:22.996 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":398085786125260623},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004635","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:21:59.817702Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000780","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:25:50.349364Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002374","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:21:28.930530Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003569","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:29:17.796678Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T06:29:22.997 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-02-11T06:29:22.997 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:29:22.997 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:29:22.997 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:29:23.081 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:29:23.081 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:29:23.339 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"00000000000000000000:00000000000000004635","last_update":"2021-02-11T06:21:59.817702Z"},{"marker":"00000000000000000000:00000000000000000780","last_update":"2021-02-11T06:25:50.349364Z"},{"marker":"00000000000000000000:00000000000000002374","last_update":"2021-02-11T06:21:28.930530Z"},{"marker":"00000000000000000000:00000000000000003569","last_update":"2021-02-11T06:29:17.796678Z"}] 2021-02-11T06:29:23.339 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-02-11T06:29:23.339 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:29:23.340 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:29:23.340 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:29:23.383 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:29:23.384 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:29:23.652 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":11837782300501369896},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004635","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:21:59.817702Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000780","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:25:50.349364Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002374","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:21:28.930530Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003569","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:29:17.796678Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T06:29:23.652 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-02-11T06:29:23.653 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:29:23.653 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:29:23.653 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:29:23.690 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:29:23.690 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:29:23.963 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"00000000000000000000:00000000000000004635","last_update":"2021-02-11T06:21:59.639441Z"},{"marker":"00000000000000000000:00000000000000000780","last_update":"2021-02-11T06:25:50.534339Z"},{"marker":"00000000000000000000:00000000000000002507","last_update":"2021-02-11T06:21:37.634942Z"},{"marker":"00000000000000000000:00000000000000003569","last_update":"2021-02-11T06:29:17.877002Z"}] 2021-02-11T06:29:23.963 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:29:23.964 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:29:23.964 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T06:29:24.039 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:29:24.040 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:29:24.305 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:29:24.306 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:29:24.306 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:29:24.342 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:29:24.343 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:29:24.609 INFO:teuthology.orchestra.run.smithi172.stdout:[{"marker":"00000000000000000000:00000000000000004635","last_update":"2021-02-11T06:21:59.791157Z"},{"marker":"00000000000000000000:00000000000000000780","last_update":"2021-02-11T06:25:50.571878Z"},{"marker":"00000000000000000000:00000000000000002374","last_update":"2021-02-11T06:21:28.667313Z"},{"marker":"00000000000000000000:00000000000000003569","last_update":"2021-02-11T06:29:17.830216Z"}] 2021-02-11T06:29:24.609 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:29:24.609 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:29:24.610 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T06:29:24.686 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:29:24.686 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:29:24.957 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:29:24.957 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:29:24.957 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:29:24.994 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:29:24.995 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:29:25.281 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"00000000000000000000:00000000000000004635","last_update":"2021-02-11T06:21:59.639441Z"},{"marker":"00000000000000000000:00000000000000000780","last_update":"2021-02-11T06:25:50.534339Z"},{"marker":"00000000000000000000:00000000000000002507","last_update":"2021-02-11T06:21:37.634942Z"},{"marker":"00000000000000000000:00000000000000003569","last_update":"2021-02-11T06:29:17.877002Z"}] 2021-02-11T06:29:25.281 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:29:25.282 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:29:25.282 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T06:29:25.361 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:29:25.361 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:29:25.647 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_datalog_autotrim ... ok 2021-02-11T06:29:25.648 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:29:25.648 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:29:25.648 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --sync-from-all=false --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T06:29:25.686 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:29:25.687 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:29:25.715 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:29:25.708+0000 7f34e7f71080 0 failed reading obj info from .rgw.root:zone_info.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7: (2) No such file or directory 2021-02-11T06:29:25.715 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:29:25.708+0000 7f34e7f71080 0 WARNING: could not read zone params for zone id=5f91a5b1-d54f-4a3d-a100-b2911a08a2e7 name=test-zone1 2021-02-11T06:29:25.716 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:29:25.708+0000 7f34e7f71080 0 failed reading obj info from .rgw.root:zone_info.bca56477-0808-4ebe-8649-b1169fc76383: (2) No such file or directory 2021-02-11T06:29:25.716 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:29:25.708+0000 7f34e7f71080 0 WARNING: could not read zone params for zone id=bca56477-0808-4ebe-8649-b1169fc76383 name=test-zone3 2021-02-11T06:29:25.733 INFO:teuthology.orchestra.run.smithi172.stdout:{"id":"e4b3955d-973b-4dd2-ae54-b7e681fbeb21","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"307AEWIWKOCPYEOH","secret_key":"qEUy5RJVuZgF4tphJ6ujQNFJaaIFkUpT"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"a797f55e-5ffc-4df8-acfc-1a39dd991822","notif_pool":"test-zone2.rgw.log:notif"} 2021-02-11T06:29:25.733 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:29:25.733 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:29:25.734 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T06:29:25.814 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:29:25.814 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:29:26.085 INFO:teuthology.orchestra.run.smithi172.stderr:Sending period to new master zone 5f91a5b1-d54f-4a3d-a100-b2911a08a2e7 2021-02-11T06:29:26.212 INFO:teuthology.orchestra.run.smithi172.stdout:{"id":"0d061ff0-ede8-4baa-beef-d975d317d505","epoch":2,"predecessor_uuid":"2afe6ddd-1cf8-433a-ae51-67de25931f7b","sync_status":["1_1613024818.417656_271.1","","",""],"period_map":{"id":"0d061ff0-ede8-4baa-beef-d975d317d505","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi061.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7","zones":[{"id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7","name":"test-zone1","endpoints":["http://smithi061.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"bca56477-0808-4ebe-8649-b1169fc76383","name":"test-zone3","endpoints":["http://smithi061.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"d7ed052a-b089-4bac-932c-80450ae3dd3d","name":"test-zone4","endpoints":["http://smithi172.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"e4b3955d-973b-4dd2-ae54-b7e681fbeb21","name":"test-zone2","endpoints":["http://smithi172.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"false","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"a797f55e-5ffc-4df8-acfc-1a39dd991822","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7","val":3877615972},{"key":"bca56477-0808-4ebe-8649-b1169fc76383","val":2043214290},{"key":"d7ed052a-b089-4bac-932c-80450ae3dd3d","val":3210761346},{"key":"e4b3955d-973b-4dd2-ae54-b7e681fbeb21","val":3820998694}]},"master_zonegroup":"test-zonegroup","master_zone":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"a797f55e-5ffc-4df8-acfc-1a39dd991822","realm_name":"test-realm","realm_epoch":4} 2021-02-11T06:29:26.212 INFO:rgw_multi.tests:Set sync_from_all flag on zone test-zone2 to false 2021-02-11T06:30:26.213 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xscaju-49 2021-02-11T06:30:26.303 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:30:26.303 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:30:26.303 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:30:26.341 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:30:26.341 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:30:26.577 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=0d061ff0-ede8-4baa-beef-d975d317d505 2021-02-11T06:30:26.614 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613025026.253230_121.1","last_update":"2021-02-11T06:30:26.253230Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:30:26.614 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T06:30:26.615 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:30:26.615 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:30:26.615 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:30:26.654 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:30:26.654 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:30:26.916 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613024957.411871_109.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:29:17.411871Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:30:26.916 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613025026.253230_121.1 target=1_1613024957.411871_109.1 2021-02-11T06:30:31.916 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:30:31.917 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:30:31.917 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:30:31.958 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:30:31.958 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:30:32.232 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613024957.411871_109.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:29:17.411871Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:30:32.232 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613025026.253230_121.1 target=1_1613024957.411871_109.1 2021-02-11T06:30:37.233 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:30:44.264 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:30:44.264 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:30:44.301 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:30:44.301 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:30:44.566 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613024957.411871_109.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:29:17.411871Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:30:44.566 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613025026.253230_121.1 target=1_1613024957.411871_109.1 2021-02-11T06:30:49.567 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:30:49.568 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:30:49.568 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:30:49.610 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:30:49.610 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:30:49.881 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613024957.411871_109.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:29:17.411871Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:30:49.882 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613025026.253230_121.1 target=1_1613024957.411871_109.1 2021-02-11T06:30:54.883 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:31:04.455 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:31:04.456 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:31:04.505 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:31:04.505 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:31:04.764 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613024957.411871_109.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:29:17.411871Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:31:04.764 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613025026.253230_121.1 target=1_1613024957.411871_109.1 2021-02-11T06:31:09.765 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:31:09.766 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:31:09.766 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:31:09.807 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:31:09.807 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:31:10.067 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613024957.411871_109.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:29:17.411871Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:31:10.067 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613025026.253230_121.1 target=1_1613024957.411871_109.1 2021-02-11T06:31:15.068 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:31:19.991 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:31:19.991 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:31:20.033 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:31:20.033 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:31:20.295 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613024957.411871_109.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:29:17.411871Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:31:20.295 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613025026.253230_121.1 target=1_1613024957.411871_109.1 2021-02-11T06:31:25.296 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:31:25.296 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:31:25.296 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:31:25.336 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:31:25.336 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:31:25.603 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613024957.411871_109.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:29:17.411871Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:31:25.603 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613025026.253230_121.1 target=1_1613024957.411871_109.1 2021-02-11T06:31:26.409 INFO:tasks.rgw.c1.client.1.smithi061.stdout:2021-02-11T06:31:26.406+0000 7f701d7f2700 -1 data sync zone:5f91a5b1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T06:31:26.415 INFO:tasks.rgw.c2.client.1.smithi172.stdout:2021-02-11T06:31:26.409+0000 7f2bc6ffd700 -1 data sync zone:5f91a5b1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T06:31:30.605 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:31:30.605 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:31:30.605 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:31:30.646 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:31:30.646 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:31:30.914 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613025026.253230_121.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:30:26.253230Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:31:30.914 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T06:31:30.914 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T06:31:30.914 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:31:30.914 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:31:30.915 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:31:30.954 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:31:30.954 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:31:31.201 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613025026.253230_121.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:30:26.253230Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:31:31.202 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T06:31:31.202 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:31:31.202 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:31:31.202 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:31:31.202 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:31:31.247 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:31:31.248 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:31:31.506 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613025026.253230_121.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:30:26.253230Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:31:31.506 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:31:31.524 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=5f91a5b1-d54f-4a3d-a100-b2911a08a2e7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:31:31.524 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=5f91a5b1-d54f-4a3d-a100-b2911a08a2e7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:31:31.524 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --redirect-zone=5f91a5b1-d54f-4a3d-a100-b2911a08a2e7 --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T06:31:31.560 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:31:31.560 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:31:31.587 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:31:31.581+0000 7f8be9306080 0 failed reading obj info from .rgw.root:zone_info.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7: (2) No such file or directory 2021-02-11T06:31:31.587 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:31:31.581+0000 7f8be9306080 0 WARNING: could not read zone params for zone id=5f91a5b1-d54f-4a3d-a100-b2911a08a2e7 name=test-zone1 2021-02-11T06:31:31.588 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:31:31.581+0000 7f8be9306080 0 failed reading obj info from .rgw.root:zone_info.bca56477-0808-4ebe-8649-b1169fc76383: (2) No such file or directory 2021-02-11T06:31:31.588 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:31:31.581+0000 7f8be9306080 0 WARNING: could not read zone params for zone id=bca56477-0808-4ebe-8649-b1169fc76383 name=test-zone3 2021-02-11T06:31:31.604 INFO:teuthology.orchestra.run.smithi172.stdout:{"id":"e4b3955d-973b-4dd2-ae54-b7e681fbeb21","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"307AEWIWKOCPYEOH","secret_key":"qEUy5RJVuZgF4tphJ6ujQNFJaaIFkUpT"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"a797f55e-5ffc-4df8-acfc-1a39dd991822","notif_pool":"test-zone2.rgw.log:notif"} 2021-02-11T06:31:31.604 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:31:31.604 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:31:31.605 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T06:31:31.687 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:31:31.687 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:31:31.955 INFO:teuthology.orchestra.run.smithi172.stderr:Sending period to new master zone 5f91a5b1-d54f-4a3d-a100-b2911a08a2e7 2021-02-11T06:31:32.052 INFO:teuthology.orchestra.run.smithi172.stdout:{"id":"0d061ff0-ede8-4baa-beef-d975d317d505","epoch":3,"predecessor_uuid":"2afe6ddd-1cf8-433a-ae51-67de25931f7b","sync_status":["1_1613024818.417656_271.1","","",""],"period_map":{"id":"0d061ff0-ede8-4baa-beef-d975d317d505","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi061.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7","zones":[{"id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7","name":"test-zone1","endpoints":["http://smithi061.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"bca56477-0808-4ebe-8649-b1169fc76383","name":"test-zone3","endpoints":["http://smithi061.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"d7ed052a-b089-4bac-932c-80450ae3dd3d","name":"test-zone4","endpoints":["http://smithi172.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"e4b3955d-973b-4dd2-ae54-b7e681fbeb21","name":"test-zone2","endpoints":["http://smithi172.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"false","sync_from":[],"redirect_zone":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7"}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"a797f55e-5ffc-4df8-acfc-1a39dd991822","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7","val":3877615972},{"key":"bca56477-0808-4ebe-8649-b1169fc76383","val":2043214290},{"key":"d7ed052a-b089-4bac-932c-80450ae3dd3d","val":3210761346},{"key":"e4b3955d-973b-4dd2-ae54-b7e681fbeb21","val":3820998694}]},"master_zonegroup":"test-zonegroup","master_zone":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"a797f55e-5ffc-4df8-acfc-1a39dd991822","realm_name":"test-realm","realm_epoch":4} 2021-02-11T06:31:32.052 INFO:rgw_multi.tests:Set redirect_zone zone test-zone2 to "5f91a5b1-d54f-4a3d-a100-b2911a08a2e7" 2021-02-11T06:32:32.415 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:32:32.415 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:32:32.416 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --sync-from-all=true --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T06:32:32.455 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:32:32.455 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:32:32.483 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:32:32.476+0000 7f04c92ad080 0 failed reading obj info from .rgw.root:zone_info.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7: (2) No such file or directory 2021-02-11T06:32:32.484 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:32:32.476+0000 7f04c92ad080 0 WARNING: could not read zone params for zone id=5f91a5b1-d54f-4a3d-a100-b2911a08a2e7 name=test-zone1 2021-02-11T06:32:32.484 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:32:32.476+0000 7f04c92ad080 0 failed reading obj info from .rgw.root:zone_info.bca56477-0808-4ebe-8649-b1169fc76383: (2) No such file or directory 2021-02-11T06:32:32.484 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:32:32.476+0000 7f04c92ad080 0 WARNING: could not read zone params for zone id=bca56477-0808-4ebe-8649-b1169fc76383 name=test-zone3 2021-02-11T06:32:32.500 INFO:teuthology.orchestra.run.smithi172.stdout:{"id":"e4b3955d-973b-4dd2-ae54-b7e681fbeb21","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"307AEWIWKOCPYEOH","secret_key":"qEUy5RJVuZgF4tphJ6ujQNFJaaIFkUpT"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"a797f55e-5ffc-4df8-acfc-1a39dd991822","notif_pool":"test-zone2.rgw.log:notif"} 2021-02-11T06:32:32.501 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:32:32.501 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:32:32.501 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T06:32:32.581 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:32:32.582 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:32:32.845 INFO:teuthology.orchestra.run.smithi172.stderr:Sending period to new master zone 5f91a5b1-d54f-4a3d-a100-b2911a08a2e7 2021-02-11T06:32:32.949 INFO:teuthology.orchestra.run.smithi172.stdout:{"id":"0d061ff0-ede8-4baa-beef-d975d317d505","epoch":4,"predecessor_uuid":"2afe6ddd-1cf8-433a-ae51-67de25931f7b","sync_status":["1_1613024818.417656_271.1","","",""],"period_map":{"id":"0d061ff0-ede8-4baa-beef-d975d317d505","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi061.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7","zones":[{"id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7","name":"test-zone1","endpoints":["http://smithi061.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"bca56477-0808-4ebe-8649-b1169fc76383","name":"test-zone3","endpoints":["http://smithi061.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"d7ed052a-b089-4bac-932c-80450ae3dd3d","name":"test-zone4","endpoints":["http://smithi172.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"e4b3955d-973b-4dd2-ae54-b7e681fbeb21","name":"test-zone2","endpoints":["http://smithi172.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7"}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"a797f55e-5ffc-4df8-acfc-1a39dd991822","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7","val":3877615972},{"key":"bca56477-0808-4ebe-8649-b1169fc76383","val":2043214290},{"key":"d7ed052a-b089-4bac-932c-80450ae3dd3d","val":3210761346},{"key":"e4b3955d-973b-4dd2-ae54-b7e681fbeb21","val":3820998694}]},"master_zonegroup":"test-zonegroup","master_zone":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"a797f55e-5ffc-4df8-acfc-1a39dd991822","realm_name":"test-realm","realm_epoch":4} 2021-02-11T06:32:32.949 INFO:rgw_multi.tests:Set sync_from_all flag on zone test-zone2 to true 2021-02-11T06:33:32.950 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:33:32.951 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:33:32.951 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --redirect-zone= --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T06:33:32.992 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:33:32.992 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:33:33.021 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:33:33.014+0000 7f16bec27080 0 failed reading obj info from .rgw.root:zone_info.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7: (2) No such file or directory 2021-02-11T06:33:33.021 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:33:33.014+0000 7f16bec27080 0 WARNING: could not read zone params for zone id=5f91a5b1-d54f-4a3d-a100-b2911a08a2e7 name=test-zone1 2021-02-11T06:33:33.021 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:33:33.014+0000 7f16bec27080 0 failed reading obj info from .rgw.root:zone_info.bca56477-0808-4ebe-8649-b1169fc76383: (2) No such file or directory 2021-02-11T06:33:33.021 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:33:33.014+0000 7f16bec27080 0 WARNING: could not read zone params for zone id=bca56477-0808-4ebe-8649-b1169fc76383 name=test-zone3 2021-02-11T06:33:33.037 INFO:teuthology.orchestra.run.smithi172.stdout:{"id":"e4b3955d-973b-4dd2-ae54-b7e681fbeb21","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"307AEWIWKOCPYEOH","secret_key":"qEUy5RJVuZgF4tphJ6ujQNFJaaIFkUpT"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"a797f55e-5ffc-4df8-acfc-1a39dd991822","notif_pool":"test-zone2.rgw.log:notif"} 2021-02-11T06:33:33.037 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:33:33.038 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:33:33.038 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T06:33:33.115 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:33:33.115 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:33:33.375 INFO:teuthology.orchestra.run.smithi172.stderr:Sending period to new master zone 5f91a5b1-d54f-4a3d-a100-b2911a08a2e7 2021-02-11T06:33:33.474 INFO:teuthology.orchestra.run.smithi172.stdout:{"id":"0d061ff0-ede8-4baa-beef-d975d317d505","epoch":5,"predecessor_uuid":"2afe6ddd-1cf8-433a-ae51-67de25931f7b","sync_status":["1_1613024818.417656_271.1","","",""],"period_map":{"id":"0d061ff0-ede8-4baa-beef-d975d317d505","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi061.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7","zones":[{"id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7","name":"test-zone1","endpoints":["http://smithi061.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"bca56477-0808-4ebe-8649-b1169fc76383","name":"test-zone3","endpoints":["http://smithi061.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"d7ed052a-b089-4bac-932c-80450ae3dd3d","name":"test-zone4","endpoints":["http://smithi172.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"e4b3955d-973b-4dd2-ae54-b7e681fbeb21","name":"test-zone2","endpoints":["http://smithi172.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"a797f55e-5ffc-4df8-acfc-1a39dd991822","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7","val":3877615972},{"key":"bca56477-0808-4ebe-8649-b1169fc76383","val":2043214290},{"key":"d7ed052a-b089-4bac-932c-80450ae3dd3d","val":3210761346},{"key":"e4b3955d-973b-4dd2-ae54-b7e681fbeb21","val":3820998694}]},"master_zonegroup":"test-zonegroup","master_zone":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"a797f55e-5ffc-4df8-acfc-1a39dd991822","realm_name":"test-realm","realm_epoch":4} 2021-02-11T06:33:33.474 INFO:rgw_multi.tests:Set redirect_zone zone test-zone2 to "" 2021-02-11T06:34:33.476 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_multi_zone_redirect ... ok 2021-02-11T06:34:39.395 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:34:39.395 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:34:39.395 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:34:39.436 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:34:39.437 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:34:39.676 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=0d061ff0-ede8-4baa-beef-d975d317d505 2021-02-11T06:34:39.713 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613025026.253230_121.1","last_update":"2021-02-11T06:30:26.253230Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:34:39.714 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T06:34:39.714 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:34:39.714 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:34:39.714 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:34:39.755 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:34:39.755 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:34:40.014 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613025026.253230_121.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:30:26.253230Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:34:40.015 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T06:34:40.015 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T06:34:40.015 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:34:40.015 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:34:40.015 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:34:40.056 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:34:40.056 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:34:40.313 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613025026.253230_121.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:30:26.253230Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:34:40.314 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T06:34:40.314 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:34:40.314 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:34:40.314 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:34:40.314 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:34:40.350 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:34:40.351 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:34:40.603 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613025026.253230_121.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:30:26.253230Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:34:40.604 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:34:40.604 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', '307AEWIWKOCPYEOH', '--secret', 'qEUy5RJVuZgF4tphJ6ujQNFJaaIFkUpT', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:34:40.604 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', '307AEWIWKOCPYEOH', '--secret', 'qEUy5RJVuZgF4tphJ6ujQNFJaaIFkUpT', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:34:40.604 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key 307AEWIWKOCPYEOH --secret qEUy5RJVuZgF4tphJ6ujQNFJaaIFkUpT --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T06:34:40.688 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:34:40.689 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:34:40.757 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:34:40.748+0000 7f4bb9852080 0 failed reading obj info from .rgw.root:zone_info.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7: (2) No such file or directory 2021-02-11T06:34:40.757 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:34:40.748+0000 7f4bb9852080 0 WARNING: could not read zone params for zone id=5f91a5b1-d54f-4a3d-a100-b2911a08a2e7 name=test-zone1 2021-02-11T06:34:40.758 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:34:40.752+0000 7f4bb9852080 0 failed reading obj info from .rgw.root:zone_info.bca56477-0808-4ebe-8649-b1169fc76383: (2) No such file or directory 2021-02-11T06:34:40.758 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:34:40.752+0000 7f4bb9852080 0 WARNING: could not read zone params for zone id=bca56477-0808-4ebe-8649-b1169fc76383 name=test-zone3 2021-02-11T06:34:40.773 INFO:teuthology.orchestra.run.smithi172.stdout:{"id":"8637dfeb-d7eb-45b8-9735-f5eaae3b3710","name":"remove","domain_root":"remove.rgw.meta:root","control_pool":"remove.rgw.control","gc_pool":"remove.rgw.log:gc","lc_pool":"remove.rgw.log:lc","log_pool":"remove.rgw.log","intent_log_pool":"remove.rgw.log:intent","usage_log_pool":"remove.rgw.log:usage","roles_pool":"remove.rgw.meta:roles","reshard_pool":"remove.rgw.log:reshard","user_keys_pool":"remove.rgw.meta:users.keys","user_email_pool":"remove.rgw.meta:users.email","user_swift_pool":"remove.rgw.meta:users.swift","user_uid_pool":"remove.rgw.meta:users.uid","otp_pool":"remove.rgw.otp","system_key":{"access_key":"307AEWIWKOCPYEOH","secret_key":"qEUy5RJVuZgF4tphJ6ujQNFJaaIFkUpT"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"remove.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"remove.rgw.buckets.data"}},"data_extra_pool":"remove.rgw.buckets.non-ec","index_type":0}}],"realm_id":"a797f55e-5ffc-4df8-acfc-1a39dd991822","notif_pool":"remove.rgw.log:notif"} 2021-02-11T06:34:40.773 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:34:40.773 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:34:40.774 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T06:34:40.858 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:34:40.858 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:34:48.447 INFO:teuthology.orchestra.run.smithi172.stderr:Sending period to new master zone 5f91a5b1-d54f-4a3d-a100-b2911a08a2e7 2021-02-11T06:35:07.657 INFO:teuthology.orchestra.run.smithi172.stdout:{"id":"0d061ff0-ede8-4baa-beef-d975d317d505","epoch":6,"predecessor_uuid":"2afe6ddd-1cf8-433a-ae51-67de25931f7b","sync_status":["1_1613024818.417656_271.1","","",""],"period_map":{"id":"0d061ff0-ede8-4baa-beef-d975d317d505","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi061.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7","zones":[{"id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7","name":"test-zone1","endpoints":["http://smithi061.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"8637dfeb-d7eb-45b8-9735-f5eaae3b3710","name":"remove","endpoints":[],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"bca56477-0808-4ebe-8649-b1169fc76383","name":"test-zone3","endpoints":["http://smithi061.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"d7ed052a-b089-4bac-932c-80450ae3dd3d","name":"test-zone4","endpoints":["http://smithi172.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"e4b3955d-973b-4dd2-ae54-b7e681fbeb21","name":"test-zone2","endpoints":["http://smithi172.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"a797f55e-5ffc-4df8-acfc-1a39dd991822","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7","val":3877615972},{"key":"8637dfeb-d7eb-45b8-9735-f5eaae3b3710","val":3690003695},{"key":"bca56477-0808-4ebe-8649-b1169fc76383","val":2043214290},{"key":"d7ed052a-b089-4bac-932c-80450ae3dd3d","val":3210761346},{"key":"e4b3955d-973b-4dd2-ae54-b7e681fbeb21","val":3820998694}]},"master_zonegroup":"test-zonegroup","master_zone":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"a797f55e-5ffc-4df8-acfc-1a39dd991822","realm_name":"test-realm","realm_epoch":4} 2021-02-11T06:35:07.658 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:35:07.658 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:35:07.658 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T06:35:07.706 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:35:07.706 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:35:07.734 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:35:07.730+0000 7feda5d6a080 0 failed reading obj info from .rgw.root:zone_info.d7ed052a-b089-4bac-932c-80450ae3dd3d: (2) No such file or directory 2021-02-11T06:35:07.734 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:35:07.730+0000 7feda5d6a080 0 WARNING: could not read zone params for zone id=d7ed052a-b089-4bac-932c-80450ae3dd3d name=test-zone4 2021-02-11T06:35:07.735 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:35:07.730+0000 7feda5d6a080 0 failed reading obj info from .rgw.root:zone_info.e4b3955d-973b-4dd2-ae54-b7e681fbeb21: (2) No such file or directory 2021-02-11T06:35:07.736 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:35:07.730+0000 7feda5d6a080 0 WARNING: could not read zone params for zone id=e4b3955d-973b-4dd2-ae54-b7e681fbeb21 name=test-zone2 2021-02-11T06:35:07.746 INFO:teuthology.orchestra.run.smithi061.stdout:{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi061.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7","zones":[{"id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7","name":"test-zone1","endpoints":["http://smithi061.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"bca56477-0808-4ebe-8649-b1169fc76383","name":"test-zone3","endpoints":["http://smithi061.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"d7ed052a-b089-4bac-932c-80450ae3dd3d","name":"test-zone4","endpoints":["http://smithi172.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"e4b3955d-973b-4dd2-ae54-b7e681fbeb21","name":"test-zone2","endpoints":["http://smithi172.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"a797f55e-5ffc-4df8-acfc-1a39dd991822","sync_policy":{"groups":[]}} 2021-02-11T06:35:07.747 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:35:07.747 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:35:07.747 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T06:35:07.830 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:35:07.830 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:35:07.853 INFO:teuthology.orchestra.run.smithi061.stderr:zone name remove not found in zonegroup test-zonegroup 2021-02-11T06:35:07.860 DEBUG:teuthology.orchestra.run:got remote process result: 2 2021-02-11T06:35:07.860 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:35:07.860 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:35:07.860 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone delete --rgw-zone remove --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T06:35:07.913 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:35:07.913 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:35:07.964 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:35:07.956+0000 7fd3d80fc080 0 failed reading obj info from .rgw.root:zone_info.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7: (2) No such file or directory 2021-02-11T06:35:07.965 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:35:07.956+0000 7fd3d80fc080 0 WARNING: could not read zone params for zone id=5f91a5b1-d54f-4a3d-a100-b2911a08a2e7 name=test-zone1 2021-02-11T06:35:07.965 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:35:07.956+0000 7fd3d80fc080 0 failed reading obj info from .rgw.root:zone_info.bca56477-0808-4ebe-8649-b1169fc76383: (2) No such file or directory 2021-02-11T06:35:07.965 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:35:07.956+0000 7fd3d80fc080 0 WARNING: could not read zone params for zone id=bca56477-0808-4ebe-8649-b1169fc76383 name=test-zone3 2021-02-11T06:35:07.970 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:35:07.960+0000 7fd3d80fc080 0 zone id 8637dfeb-d7eb-45b8-9735-f5eaae3b3710 is not a part of zonegroup default 2021-02-11T06:35:07.985 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:35:07.986 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:35:07.986 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T06:35:08.029 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:35:08.029 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:35:08.103 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:35:08.098+0000 7f83835aa080 0 WARNING: can't generate connection for zone 8637dfeb-d7eb-45b8-9735-f5eaae3b3710 id remove: no endpoints defined 2021-02-11T06:35:08.365 INFO:teuthology.orchestra.run.smithi061.stdout:{"id":"0d061ff0-ede8-4baa-beef-d975d317d505","epoch":7,"predecessor_uuid":"2afe6ddd-1cf8-433a-ae51-67de25931f7b","sync_status":["1_1613024818.417656_271.1","","",""],"period_map":{"id":"0d061ff0-ede8-4baa-beef-d975d317d505","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi061.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7","zones":[{"id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7","name":"test-zone1","endpoints":["http://smithi061.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"bca56477-0808-4ebe-8649-b1169fc76383","name":"test-zone3","endpoints":["http://smithi061.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"d7ed052a-b089-4bac-932c-80450ae3dd3d","name":"test-zone4","endpoints":["http://smithi172.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"e4b3955d-973b-4dd2-ae54-b7e681fbeb21","name":"test-zone2","endpoints":["http://smithi172.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"a797f55e-5ffc-4df8-acfc-1a39dd991822","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7","val":3877615972},{"key":"bca56477-0808-4ebe-8649-b1169fc76383","val":2043214290},{"key":"d7ed052a-b089-4bac-932c-80450ae3dd3d","val":3210761346},{"key":"e4b3955d-973b-4dd2-ae54-b7e681fbeb21","val":3820998694}]},"master_zonegroup":"test-zonegroup","master_zone":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"a797f55e-5ffc-4df8-acfc-1a39dd991822","realm_name":"test-realm","realm_epoch":4} 2021-02-11T06:35:08.365 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_zonegroup_remove ... ok 2021-02-11T06:35:08.366 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:35:08.366 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:35:08.366 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup create --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T06:35:08.447 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:35:08.447 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:35:08.480 INFO:teuthology.orchestra.run.smithi061.stdout:{"id":"c6968d4c-0ae2-43a0-987b-c10544a59c5e","name":"remove_zg","api_name":"remove_zg","is_master":"false","endpoints":[],"hostnames":[],"hostnames_s3website":[],"master_zone":"","zones":[],"placement_targets":[],"default_placement":"","realm_id":"a797f55e-5ffc-4df8-acfc-1a39dd991822","sync_policy":{"groups":[]}} 2021-02-11T06:35:08.481 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:35:08.481 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:35:08.481 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T06:35:08.563 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:35:08.564 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:35:08.629 INFO:teuthology.orchestra.run.smithi061.stdout:{"id":"6227e28f-ef4c-4af7-9fcc-a9a08bc689ab","name":"remove","domain_root":"remove.rgw.meta:root","control_pool":"remove.rgw.control","gc_pool":"remove.rgw.log:gc","lc_pool":"remove.rgw.log:lc","log_pool":"remove.rgw.log","intent_log_pool":"remove.rgw.log:intent","usage_log_pool":"remove.rgw.log:usage","roles_pool":"remove.rgw.meta:roles","reshard_pool":"remove.rgw.log:reshard","user_keys_pool":"remove.rgw.meta:users.keys","user_email_pool":"remove.rgw.meta:users.email","user_swift_pool":"remove.rgw.meta:users.swift","user_uid_pool":"remove.rgw.meta:users.uid","otp_pool":"remove.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"remove.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"remove.rgw.buckets.data"}},"data_extra_pool":"remove.rgw.buckets.non-ec","index_type":0}}],"realm_id":"a797f55e-5ffc-4df8-acfc-1a39dd991822","notif_pool":"remove.rgw.log:notif"} 2021-02-11T06:35:08.630 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:35:08.630 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:35:08.630 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T06:35:08.720 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:35:08.720 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:35:09.139 INFO:teuthology.orchestra.run.smithi061.stdout:{"id":"0d061ff0-ede8-4baa-beef-d975d317d505","epoch":8,"predecessor_uuid":"2afe6ddd-1cf8-433a-ae51-67de25931f7b","sync_status":["1_1613024818.417656_271.1","","",""],"period_map":{"id":"0d061ff0-ede8-4baa-beef-d975d317d505","zonegroups":[{"id":"c6968d4c-0ae2-43a0-987b-c10544a59c5e","name":"remove_zg","api_name":"remove_zg","is_master":"false","endpoints":[],"hostnames":[],"hostnames_s3website":[],"master_zone":"6227e28f-ef4c-4af7-9fcc-a9a08bc689ab","zones":[{"id":"6227e28f-ef4c-4af7-9fcc-a9a08bc689ab","name":"remove","endpoints":[],"log_meta":"false","log_data":"false","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"a797f55e-5ffc-4df8-acfc-1a39dd991822","sync_policy":{"groups":[]}},{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi061.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7","zones":[{"id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7","name":"test-zone1","endpoints":["http://smithi061.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"bca56477-0808-4ebe-8649-b1169fc76383","name":"test-zone3","endpoints":["http://smithi061.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"d7ed052a-b089-4bac-932c-80450ae3dd3d","name":"test-zone4","endpoints":["http://smithi172.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"e4b3955d-973b-4dd2-ae54-b7e681fbeb21","name":"test-zone2","endpoints":["http://smithi172.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"a797f55e-5ffc-4df8-acfc-1a39dd991822","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7","val":3877615972},{"key":"6227e28f-ef4c-4af7-9fcc-a9a08bc689ab","val":995540807},{"key":"bca56477-0808-4ebe-8649-b1169fc76383","val":2043214290},{"key":"d7ed052a-b089-4bac-932c-80450ae3dd3d","val":3210761346},{"key":"e4b3955d-973b-4dd2-ae54-b7e681fbeb21","val":3820998694}]},"master_zonegroup":"test-zonegroup","master_zone":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"a797f55e-5ffc-4df8-acfc-1a39dd991822","realm_name":"test-realm","realm_epoch":4} 2021-02-11T06:35:09.139 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:35:09.139 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:35:09.140 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone delete --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T06:35:09.220 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:35:09.221 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:35:09.267 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:35:09.262+0000 7f902c605080 0 zone id 6227e28f-ef4c-4af7-9fcc-a9a08bc689ab is not a part of zonegroup test-zonegroup 2021-02-11T06:35:09.268 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:35:09.262+0000 7f902c605080 0 zone id 6227e28f-ef4c-4af7-9fcc-a9a08bc689ab is not a part of zonegroup default 2021-02-11T06:35:09.282 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:35:09.282 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:35:09.282 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T06:35:09.367 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:35:09.368 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:35:09.412 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:35:09.406+0000 7f8e25efe080 0 ERROR: zonegroup remove_zg has a non existent master zone 2021-02-11T06:35:09.418 DEBUG:teuthology.orchestra.run:got remote process result: 22 2021-02-11T06:35:09.418 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:35:09.418 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:35:09.418 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup delete --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T06:35:09.496 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:35:09.496 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:35:09.533 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:35:09.533 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:35:09.534 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T06:35:09.619 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:35:09.619 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:35:09.951 INFO:teuthology.orchestra.run.smithi061.stdout:{"id":"0d061ff0-ede8-4baa-beef-d975d317d505","epoch":9,"predecessor_uuid":"2afe6ddd-1cf8-433a-ae51-67de25931f7b","sync_status":["1_1613024818.417656_271.1","","",""],"period_map":{"id":"0d061ff0-ede8-4baa-beef-d975d317d505","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi061.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7","zones":[{"id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7","name":"test-zone1","endpoints":["http://smithi061.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"bca56477-0808-4ebe-8649-b1169fc76383","name":"test-zone3","endpoints":["http://smithi061.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"d7ed052a-b089-4bac-932c-80450ae3dd3d","name":"test-zone4","endpoints":["http://smithi172.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"e4b3955d-973b-4dd2-ae54-b7e681fbeb21","name":"test-zone2","endpoints":["http://smithi172.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"a797f55e-5ffc-4df8-acfc-1a39dd991822","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7","val":3877615972},{"key":"bca56477-0808-4ebe-8649-b1169fc76383","val":2043214290},{"key":"d7ed052a-b089-4bac-932c-80450ae3dd3d","val":3210761346},{"key":"e4b3955d-973b-4dd2-ae54-b7e681fbeb21","val":3820998694}]},"master_zonegroup":"test-zonegroup","master_zone":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"a797f55e-5ffc-4df8-acfc-1a39dd991822","realm_name":"test-realm","realm_epoch":4} 2021-02-11T06:35:09.951 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_zg_master_zone_delete ... ok 2021-02-11T06:35:09.952 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xscaju-50 2021-02-11T06:35:11.138 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xscaju-51 2021-02-11T06:35:11.468 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xscaju-52 2021-02-11T06:35:11.564 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xscaju-53 2021-02-11T06:35:11.610 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xscaju-54 2021-02-11T06:35:11.682 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xscaju-55 2021-02-11T06:35:13.982 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_set_bucket_website ... tasks.rgw_multi.tests.test_set_bucket_policy ... ok 2021-02-11T06:35:13.983 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xscaju-56 2021-02-11T06:35:14.026 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xscaju-57 2021-02-11T06:35:14.097 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xscaju-58 2021-02-11T06:35:14.164 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:35:14.164 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:35:14.164 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:35:14.202 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:35:14.203 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:35:14.436 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=0d061ff0-ede8-4baa-beef-d975d317d505 2021-02-11T06:35:14.468 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613025314.142385_220.1","last_update":"2021-02-11T06:35:14.142385Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:35:14.469 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T06:35:14.469 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:35:14.469 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:35:14.469 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:35:14.509 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:35:14.509 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:35:14.780 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613025026.253230_121.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:30:26.253230Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:35:14.780 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613025314.142385_220.1 target=1_1613025026.253230_121.1 2021-02-11T06:35:19.781 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:35:32.491 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:35:32.491 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:35:32.532 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:35:32.532 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:35:32.802 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613025026.253230_121.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:30:26.253230Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:35:32.802 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613025314.142385_220.1 target=1_1613025026.253230_121.1 2021-02-11T06:35:37.803 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:35:46.521 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:35:46.521 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:35:46.564 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:35:46.565 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:35:46.831 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613025026.253230_121.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:30:26.253230Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:35:46.831 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613025314.142385_220.1 target=1_1613025026.253230_121.1 2021-02-11T06:35:51.832 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:35:51.833 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:35:51.833 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:35:51.873 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:35:51.873 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:35:52.132 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613025026.253230_121.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:30:26.253230Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:35:52.133 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613025314.142385_220.1 target=1_1613025026.253230_121.1 2021-02-11T06:35:57.133 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:35:57.134 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:35:57.134 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:35:57.175 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:35:57.176 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:35:57.432 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613025026.253230_121.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:30:26.253230Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:35:57.432 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613025314.142385_220.1 target=1_1613025026.253230_121.1 2021-02-11T06:36:02.433 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:36:25.740 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:36:25.740 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:36:25.781 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:36:25.782 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:36:26.049 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613025026.253230_121.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:30:26.253230Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:36:26.050 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613025314.142385_220.1 target=1_1613025026.253230_121.1 2021-02-11T06:36:31.051 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:36:37.607 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:36:37.607 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:36:37.647 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:36:37.648 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:36:37.910 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613025026.253230_121.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:30:26.253230Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:36:37.910 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613025314.142385_220.1 target=1_1613025026.253230_121.1 2021-02-11T06:36:42.911 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:36:42.911 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:36:42.912 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:36:42.953 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:36:42.954 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:36:43.214 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613025314.142385_220.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:35:14.142385Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:36:43.215 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T06:36:43.215 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T06:36:43.215 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:36:43.215 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:36:43.216 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:36:43.256 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:36:43.256 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:36:43.510 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613025314.142385_220.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:35:14.142385Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:36:43.511 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T06:36:43.511 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:36:43.511 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:36:43.511 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:36:43.512 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:36:43.552 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:36:43.552 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:36:43.818 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613025314.142385_220.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:35:14.142385Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:36:43.819 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:36:43.819 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'xscaju-56', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:36:43.819 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'xscaju-56', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:36:43.819 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket xscaju-56 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T06:36:43.861 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:36:43.862 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:36:44.201 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'xscaju-57', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:36:44.201 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'xscaju-57', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:36:44.202 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket xscaju-57 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T06:36:44.287 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:36:44.287 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:36:44.662 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'xscaju-58', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:36:44.662 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'xscaju-58', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:36:44.662 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket xscaju-58 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T06:36:44.747 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:36:44.747 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:36:45.087 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:36:45.088 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:36:45.088 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:36:45.170 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:36:45.170 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:36:45.456 INFO:teuthology.orchestra.run.smithi061.stdout:["obj_delete_at_hint.0000000078","meta.history","obj_delete_at_hint.0000000070","obj_delete_at_hint.0000000104","obj_delete_at_hint.0000000026","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-16:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.16:5","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-10:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.10:3","obj_delete_at_hint.0000000028","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.16-xscaju-16","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.35-xscaju-35","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-11:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.11:3","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.1-xscaju-1","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.37-xscaju-37","obj_delete_at_hint.0000000040","meta.full-sync.index.0","obj_delete_at_hint.0000000015","obj_delete_at_hint.0000000069","2021-02-11-06-e4b3955d-973b-4dd2-ae54-b7e681fbeb21.5034.1-xscaju-42","obj_delete_at_hint.0000000095","obj_delete_at_hint.0000000003","obj_delete_at_hint.0000000047","data.full-sync.index.e4b3955d-973b-4dd2-ae54-b7e681fbeb21.3","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.17-xscaju-17","obj_delete_at_hint.0000000035","obj_delete_at_hint.0000000037","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-25:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.25:8","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-15:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.15:1","obj_delete_at_hint.0000000024","obj_delete_at_hint.0000000000","obj_delete_at_hint.0000000031","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-12:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.12:8","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-49:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5375.1:5","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-22:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.22:8","data_log.3","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.31-xscaju-31","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-18:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.18:1","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-17:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.17:5","data.full-sync.index.e4b3955d-973b-4dd2-ae54-b7e681fbeb21.1","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-38:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.38:10","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-10:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.10:8","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.4-xscaju-4","obj_delete_at_hint.0000000076","obj_delete_at_hint.0000000113","obj_delete_at_hint.0000000005","datalog.sync-status.shard.bca56477-0808-4ebe-8649-b1169fc76383.2","obj_delete_at_hint.0000000011","obj_delete_at_hint.0000000058","obj_delete_at_hint.0000000052","data.full-sync.index.bca56477-0808-4ebe-8649-b1169fc76383.3","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-12:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.12:3","meta.log.0d061ff0-ede8-4baa-beef-d975d317d505.0","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-18:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.18:5","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-24:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.24:8","obj_delete_at_hint.0000000088","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-16:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.16:5","obj_delete_at_hint.0000000080","obj_delete_at_hint.0000000090","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-11:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.11:3","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.3-xscaju-52","obj_delete_at_hint.0000000110","data.full-sync.index.bca56477-0808-4ebe-8649-b1169fc76383.2","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-14:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.14:1","data.full-sync.index.e4b3955d-973b-4dd2-ae54-b7e681fbeb21.0","obj_delete_at_hint.0000000096","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.18-xscaju-18","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.27-xscaju-27","obj_delete_at_hint.0000000087","obj_delete_at_hint.0000000008","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-19:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.19:8","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-12:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.12:7","obj_delete_at_hint.0000000006","obj_delete_at_hint.0000000029","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5174.2-xscaju-45","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-25:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.25:8","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.11-xscaju-11","obj_delete_at_hint.0000000089","obj_delete_at_hint.0000000022","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.6-xscaju-55","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-37:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.37:10","bilog.trim","obj_delete_at_hint.0000000016","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-17:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.17:3","obj_delete_at_hint.0000000041","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-16:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.16:1","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-18:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.18:3","obj_delete_at_hint.0000000018","obj_delete_at_hint.0000000092","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.6-xscaju-6","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-19:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.19:8","data_log.0.z2Z1v9WY1zl.0","obj_delete_at_hint.0000000014","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-17:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.17:5","obj_delete_at_hint.0000000112","obj_delete_at_hint.0000000007","obj_delete_at_hint.0000000021","obj_delete_at_hint.0000000064","obj_delete_at_hint.0000000071","datalog.sync-status.shard.bca56477-0808-4ebe-8649-b1169fc76383.3","obj_delete_at_hint.0000000074","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5375.1-xscaju-49","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-47:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5174.4:8","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-36:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.36:10","obj_delete_at_hint.0000000081","obj_delete_at_hint.0000000009","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-10:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.10:7","obj_delete_at_hint.0000000121","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.2-xscaju-51","data_log.1","obj_delete_at_hint.0000000125","obj_delete_at_hint.0000000082","meta.full-sync.index.1","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.8-xscaju-57","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.36-xscaju-36","obj_delete_at_hint.0000000105","obj_delete_at_hint.0000000059","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-21:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.21:8","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-23:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.23:8","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-22:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.22:8","obj_delete_at_hint.0000000077","data_log.2.WA7SeTvwYnf.0","obj_delete_at_hint.0000000032","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.7-xscaju-7","obj_delete_at_hint.0000000053","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-12:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.12:1","obj_delete_at_hint.0000000091","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-12:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.12:3","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.13-xscaju-13","obj_delete_at_hint.0000000065","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-12:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.12:7","obj_delete_at_hint.0000000083","obj_delete_at_hint.0000000010","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-11:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.11:1","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-47:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5174.4:8","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5174.4-xscaju-47","mdlog.sync-status.shard.2","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-16:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.16:3","obj_delete_at_hint.0000000045","obj_delete_at_hint.0000000002","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.24-xscaju-24","data_log.3.KQWmhlkNDzT.0","datalog.sync-status.shard.e4b3955d-973b-4dd2-ae54-b7e681fbeb21.0","obj_delete_at_hint.0000000116","obj_delete_at_hint.0000000034","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-10:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.10:1","obj_delete_at_hint.0000000101","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.39-xscaju-39","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.29-xscaju-29","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.25-xscaju-25","mdlog.sync-status.shard.1","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5174.3-xscaju-46","obj_delete_at_hint.0000000079","obj_delete_at_hint.0000000049","obj_delete_at_hint.0000000117","obj_delete_at_hint.0000000044","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-19:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.19:5","obj_delete_at_hint.0000000066","datalog.sync-status.bca56477-0808-4ebe-8649-b1169fc76383","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.32-xscaju-32","obj_delete_at_hint.0000000068","obj_delete_at_hint.0000000085","obj_delete_at_hint.0000000073","obj_delete_at_hint.0000000038","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.21-xscaju-21","meta.full-sync.index.3","obj_delete_at_hint.0000000118","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-23:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.23:8","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.34-xscaju-34","data.full-sync.index.bca56477-0808-4ebe-8649-b1169fc76383.0","obj_delete_at_hint.0000000036","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-10:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.10:1","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.7-xscaju-56","obj_delete_at_hint.0000000103","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.3-xscaju-3","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.33-xscaju-33","mdlog.sync-status.shard.3","obj_delete_at_hint.0000000119","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.22-xscaju-22","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-49:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5375.1:5","obj_delete_at_hint.0000000098","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-20:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.20:8","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.38-xscaju-38","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-13:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.13:1","mdlog.sync-status.shard.0","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-46:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5174.3:8","2021-02-11-06-e4b3955d-973b-4dd2-ae54-b7e681fbeb21.5034.2-xscaju-43","obj_delete_at_hint.0000000027","datalog.sync-status.shard.bca56477-0808-4ebe-8649-b1169fc76383.1","mdlog.sync-status","obj_delete_at_hint.0000000019","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.30-xscaju-30","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-21:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.21:8","obj_delete_at_hint.0000000039","data_log.1.So5zDsvqX3x.0","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.10-xscaju-10","obj_delete_at_hint.0000000100","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-11:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.11:8","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-10:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.10:8","obj_delete_at_hint.0000000093","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.15-xscaju-15","obj_delete_at_hint.0000000004","obj_delete_at_hint.0000000063","obj_delete_at_hint.0000000122","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-12:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.12:8","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.1-xscaju-50","obj_delete_at_hint.0000000057","obj_delete_at_hint.0000000054","obj_delete_at_hint.0000000114","data_log.0","obj_delete_at_hint.0000000012","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-38:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.38:10","obj_delete_at_hint.0000000084","obj_delete_at_hint.0000000043","obj_delete_at_hint.0000000111","obj_delete_at_hint.0000000048","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.8-xscaju-8","obj_delete_at_hint.0000000020","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.2-xscaju-2","obj_delete_at_hint.0000000099","obj_delete_at_hint.0000000056","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-10:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.10:3","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-10:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.10:7","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-48:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5174.5:8","obj_delete_at_hint.0000000072","obj_delete_at_hint.0000000062","obj_delete_at_hint.0000000115","obj_delete_at_hint.0000000033","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.5-xscaju-54","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-24:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.24:8","datalog.sync-status.shard.e4b3955d-973b-4dd2-ae54-b7e681fbeb21.1","obj_delete_at_hint.0000000025","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-19:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.19:5","obj_delete_at_hint.0000000123","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.4-xscaju-53","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-11:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.11:8","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.9-xscaju-9","obj_delete_at_hint.0000000108","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-15:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.15:1","obj_delete_at_hint.0000000094","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.20-xscaju-20","obj_delete_at_hint.0000000060","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-16:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.16:1","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-18:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.18:5","obj_delete_at_hint.0000000109","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.19-xscaju-19","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.40-xscaju-40","datalog.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21","obj_delete_at_hint.0000000013","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-26:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.26:8","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.28-xscaju-28","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-11:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.11:7","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-12:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.12:1","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.14-xscaju-14","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-17:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.17:1","obj_delete_at_hint.0000000051","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-48:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5174.5:8","obj_delete_at_hint.0000000106","obj_delete_at_hint.0000000017","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-36:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.36:10","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-11:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.11:1","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.12-xscaju-12","datalog.sync-status.shard.e4b3955d-973b-4dd2-ae54-b7e681fbeb21.2","obj_delete_at_hint.0000000023","obj_delete_at_hint.0000000107","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-46:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5174.3:8","obj_delete_at_hint.0000000046","meta.full-sync.index.2","datalog.sync-status.shard.e4b3955d-973b-4dd2-ae54-b7e681fbeb21.3","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-18:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.18:3","data_log.2","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-26:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.26:8","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-14:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.14:1","obj_delete_at_hint.0000000067","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-17:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.17:3","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-17:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.17:1","obj_delete_at_hint.0000000086","data.full-sync.index.e4b3955d-973b-4dd2-ae54-b7e681fbeb21.2","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-13:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.13:1","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-18:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.18:1","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-20:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.20:8","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.9-xscaju-58","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-11:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.11:7","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5174.5-xscaju-48","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-37:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.37:10","data.full-sync.index.bca56477-0808-4ebe-8649-b1169fc76383.1","obj_delete_at_hint.0000000050","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.5-xscaju-5","obj_delete_at_hint.0000000102","datalog.sync-status.shard.bca56477-0808-4ebe-8649-b1169fc76383.0","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5174.1-xscaju-44","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.41-xscaju-41","obj_delete_at_hint.0000000124","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.26-xscaju-26","obj_delete_at_hint.0000000001","obj_delete_at_hint.0000000126","obj_delete_at_hint.0000000055","obj_delete_at_hint.0000000061","obj_delete_at_hint.0000000097","obj_delete_at_hint.0000000042","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-16:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.16:3","obj_delete_at_hint.0000000075","obj_delete_at_hint.0000000120","obj_delete_at_hint.0000000030","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.23-xscaju-23"] 2021-02-11T06:36:45.489 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:36:45.489 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:36:45.489 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:36:45.532 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:36:45.532 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:36:45.783 INFO:teuthology.orchestra.run.smithi172.stdout:["obj_delete_at_hint.0000000078","meta.history","obj_delete_at_hint.0000000070","obj_delete_at_hint.0000000104","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-17:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.17:5","obj_delete_at_hint.0000000026","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-16:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.16:5","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-26:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.26:8","obj_delete_at_hint.0000000028","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-10:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.10:3","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.16-xscaju-16","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.1-xscaju-1","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.37-xscaju-37","obj_delete_at_hint.0000000040","meta.full-sync.index.0","obj_delete_at_hint.0000000015","obj_delete_at_hint.0000000069","2021-02-11-06-e4b3955d-973b-4dd2-ae54-b7e681fbeb21.5034.1-xscaju-42","obj_delete_at_hint.0000000095","obj_delete_at_hint.0000000003","obj_delete_at_hint.0000000047","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.17-xscaju-17","obj_delete_at_hint.0000000035","obj_delete_at_hint.0000000037","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-15:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.15:1","obj_delete_at_hint.0000000024","obj_delete_at_hint.0000000000","obj_delete_at_hint.0000000031","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-22:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.22:8","data_log.3","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.31-xscaju-31","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-18:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.18:1","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-17:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.17:5","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-38:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.38:10","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-10:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.10:8","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.4-xscaju-4","obj_delete_at_hint.0000000076","obj_delete_at_hint.0000000113","obj_delete_at_hint.0000000005","datalog.sync-status.shard.bca56477-0808-4ebe-8649-b1169fc76383.2","obj_delete_at_hint.0000000011","obj_delete_at_hint.0000000058","obj_delete_at_hint.0000000052","data.full-sync.index.bca56477-0808-4ebe-8649-b1169fc76383.3","datalog.sync-status.shard.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.0","meta.log.0d061ff0-ede8-4baa-beef-d975d317d505.0","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-18:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.18:5","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-24:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.24:8","obj_delete_at_hint.0000000088","obj_delete_at_hint.0000000080","obj_delete_at_hint.0000000090","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-11:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.11:3","obj_delete_at_hint.0000000110","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-10:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.10:8","data_log.2.cs6FkHiHGNb.0","data.full-sync.index.bca56477-0808-4ebe-8649-b1169fc76383.2","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-14:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.14:1","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-10:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.10:7","obj_delete_at_hint.0000000096","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.18-xscaju-18","data_log.3.U7If5/zMdsw.0","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-11:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.11:8","obj_delete_at_hint.0000000087","obj_delete_at_hint.0000000008","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-16:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.16:1","obj_delete_at_hint.0000000006","obj_delete_at_hint.0000000029","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-11:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.11:7","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5174.2-xscaju-45","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-25:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.25:8","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.11-xscaju-11","obj_delete_at_hint.0000000089","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-20:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.20:8","obj_delete_at_hint.0000000022","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-46:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5174.3:8","bilog.trim","obj_delete_at_hint.0000000016","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-17:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.17:3","obj_delete_at_hint.0000000041","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-17:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.17:3","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-16:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.16:1","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-18:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.18:3","obj_delete_at_hint.0000000018","obj_delete_at_hint.0000000092","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-22:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.22:8","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.6-xscaju-6","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-25:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.25:8","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-19:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.19:8","obj_delete_at_hint.0000000014","obj_delete_at_hint.0000000112","obj_delete_at_hint.0000000007","obj_delete_at_hint.0000000021","obj_delete_at_hint.0000000064","obj_delete_at_hint.0000000071","datalog.sync-status.shard.bca56477-0808-4ebe-8649-b1169fc76383.3","obj_delete_at_hint.0000000074","datalog.sync-status.shard.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.2","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5375.1-xscaju-49","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-36:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.36:10","obj_delete_at_hint.0000000081","obj_delete_at_hint.0000000009","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-10:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.10:7","obj_delete_at_hint.0000000121","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.2-xscaju-51","data_log.1","obj_delete_at_hint.0000000125","obj_delete_at_hint.0000000082","meta.full-sync.index.1","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.8-xscaju-57","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.36-xscaju-36","obj_delete_at_hint.0000000105","obj_delete_at_hint.0000000059","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-23:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.23:8","obj_delete_at_hint.0000000077","obj_delete_at_hint.0000000032","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.7-xscaju-7","obj_delete_at_hint.0000000053","obj_delete_at_hint.0000000091","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-12:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.12:3","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.13-xscaju-13","obj_delete_at_hint.0000000065","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-12:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.12:7","obj_delete_at_hint.0000000083","obj_delete_at_hint.0000000010","data.full-sync.index.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.3","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-47:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5174.4:8","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5174.4-xscaju-47","datalog.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-37:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.37:10","mdlog.sync-status.shard.2","obj_delete_at_hint.0000000045","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-23:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.23:8","obj_delete_at_hint.0000000002","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.24-xscaju-24","obj_delete_at_hint.0000000116","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-11:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.11:3","obj_delete_at_hint.0000000034","obj_delete_at_hint.0000000101","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.39-xscaju-39","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.25-xscaju-25","mdlog.sync-status.shard.1","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-12:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.12:1","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5174.3-xscaju-46","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-13:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.13:1","obj_delete_at_hint.0000000079","obj_delete_at_hint.0000000049","obj_delete_at_hint.0000000117","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-47:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5174.4:8","obj_delete_at_hint.0000000044","data.full-sync.index.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.2","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-19:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.19:5","obj_delete_at_hint.0000000066","datalog.sync-status.bca56477-0808-4ebe-8649-b1169fc76383","obj_delete_at_hint.0000000068","obj_delete_at_hint.0000000085","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-38:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.38:10","obj_delete_at_hint.0000000073","obj_delete_at_hint.0000000038","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.21-xscaju-21","meta.full-sync.index.3","obj_delete_at_hint.0000000118","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.34-xscaju-34","data.full-sync.index.bca56477-0808-4ebe-8649-b1169fc76383.0","obj_delete_at_hint.0000000036","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-10:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.10:1","datalog.sync-status.shard.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.3","obj_delete_at_hint.0000000103","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.3-xscaju-3","mdlog.sync-status.shard.3","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-49:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5375.1:5","obj_delete_at_hint.0000000119","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.22-xscaju-22","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-49:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5375.1:5","obj_delete_at_hint.0000000098","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-20:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.20:8","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.38-xscaju-38","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-13:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.13:1","mdlog.sync-status.shard.0","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-46:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5174.3:8","2021-02-11-06-e4b3955d-973b-4dd2-ae54-b7e681fbeb21.5034.2-xscaju-43","obj_delete_at_hint.0000000027","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-24:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.24:8","datalog.sync-status.shard.bca56477-0808-4ebe-8649-b1169fc76383.1","mdlog.sync-status","obj_delete_at_hint.0000000019","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-19:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.19:5","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-21:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.21:8","obj_delete_at_hint.0000000039","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.10-xscaju-10","obj_delete_at_hint.0000000100","datalog.sync-status.shard.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.1","obj_delete_at_hint.0000000093","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-48:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5174.5:8","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.15-xscaju-15","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-12:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.12:7","obj_delete_at_hint.0000000004","obj_delete_at_hint.0000000063","obj_delete_at_hint.0000000122","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-12:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.12:8","obj_delete_at_hint.0000000057","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-15:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.15:1","data_log.0.VgGNQ4OUDdo.0","obj_delete_at_hint.0000000054","obj_delete_at_hint.0000000114","data_log.0","obj_delete_at_hint.0000000012","obj_delete_at_hint.0000000084","obj_delete_at_hint.0000000043","obj_delete_at_hint.0000000111","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-14:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.14:1","obj_delete_at_hint.0000000048","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-19:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.19:8","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.8-xscaju-8","obj_delete_at_hint.0000000020","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.2-xscaju-2","obj_delete_at_hint.0000000099","obj_delete_at_hint.0000000056","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-10:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.10:3","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-48:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5174.5:8","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-18:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.18:3","obj_delete_at_hint.0000000072","obj_delete_at_hint.0000000062","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-18:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.18:1","obj_delete_at_hint.0000000115","obj_delete_at_hint.0000000033","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.5-xscaju-54","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-16:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.16:5","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-18:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.18:5","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-16:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.16:3","obj_delete_at_hint.0000000025","obj_delete_at_hint.0000000123","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-11:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.11:8","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.9-xscaju-9","obj_delete_at_hint.0000000108","obj_delete_at_hint.0000000094","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.20-xscaju-20","obj_delete_at_hint.0000000060","obj_delete_at_hint.0000000109","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.19-xscaju-19","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.40-xscaju-40","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-21:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.21:8","obj_delete_at_hint.0000000013","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-26:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.26:8","data.full-sync.index.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.1","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.28-xscaju-28","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-11:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.11:7","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-12:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.12:1","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.14-xscaju-14","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-17:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.17:1","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-17:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.17:1","obj_delete_at_hint.0000000051","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-11:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.11:1","obj_delete_at_hint.0000000106","obj_delete_at_hint.0000000017","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-12:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.12:8","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-11:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.11:1","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.12-xscaju-12","obj_delete_at_hint.0000000023","obj_delete_at_hint.0000000107","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-10:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.10:1","obj_delete_at_hint.0000000046","meta.full-sync.index.2","data_log.2","obj_delete_at_hint.0000000067","data.full-sync.index.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.0","obj_delete_at_hint.0000000086","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5174.5-xscaju-48","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-37:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.37:10","data.full-sync.index.bca56477-0808-4ebe-8649-b1169fc76383.1","obj_delete_at_hint.0000000050","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.5-xscaju-5","obj_delete_at_hint.0000000102","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-12:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.12:3","datalog.sync-status.shard.bca56477-0808-4ebe-8649-b1169fc76383.0","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5174.1-xscaju-44","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.41-xscaju-41","obj_delete_at_hint.0000000124","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.26-xscaju-26","obj_delete_at_hint.0000000001","data_log.1.pkR88vyVvkW.0","obj_delete_at_hint.0000000126","obj_delete_at_hint.0000000055","obj_delete_at_hint.0000000061","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-36:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.36:10","obj_delete_at_hint.0000000097","obj_delete_at_hint.0000000042","bucket.sync-status.bca56477-0808-4ebe-8649-b1169fc76383:xscaju-16:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.16:3","obj_delete_at_hint.0000000075","obj_delete_at_hint.0000000120","obj_delete_at_hint.0000000030","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.23-xscaju-23"] 2021-02-11T06:36:45.807 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:36:45.807 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:36:45.808 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:36:45.849 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:36:45.849 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:36:46.108 INFO:teuthology.orchestra.run.smithi061.stdout:["obj_delete_at_hint.0000000078","meta.history","obj_delete_at_hint.0000000070","obj_delete_at_hint.0000000104","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-17:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.17:5","obj_delete_at_hint.0000000026","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-10:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.10:3","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-26:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.26:8","obj_delete_at_hint.0000000028","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-10:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.10:3","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.16-xscaju-16","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.35-xscaju-35","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-11:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.11:3","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.1-xscaju-1","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.37-xscaju-37","obj_delete_at_hint.0000000040","meta.full-sync.index.0","obj_delete_at_hint.0000000015","obj_delete_at_hint.0000000069","2021-02-11-06-e4b3955d-973b-4dd2-ae54-b7e681fbeb21.5034.1-xscaju-42","obj_delete_at_hint.0000000095","obj_delete_at_hint.0000000003","obj_delete_at_hint.0000000047","data.full-sync.index.e4b3955d-973b-4dd2-ae54-b7e681fbeb21.3","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.17-xscaju-17","obj_delete_at_hint.0000000035","obj_delete_at_hint.0000000037","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-25:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.25:8","obj_delete_at_hint.0000000024","obj_delete_at_hint.0000000000","obj_delete_at_hint.0000000031","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-12:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.12:8","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-49:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5375.1:5","data_log.3","data.full-sync.index.e4b3955d-973b-4dd2-ae54-b7e681fbeb21.1","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.4-xscaju-4","obj_delete_at_hint.0000000076","data_log.3.RUY6opSomey.0","obj_delete_at_hint.0000000113","obj_delete_at_hint.0000000005","obj_delete_at_hint.0000000011","obj_delete_at_hint.0000000058","obj_delete_at_hint.0000000052","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-12:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.12:3","datalog.sync-status.shard.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.0","meta.log.0d061ff0-ede8-4baa-beef-d975d317d505.0","obj_delete_at_hint.0000000088","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-16:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.16:5","obj_delete_at_hint.0000000080","obj_delete_at_hint.0000000090","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.3-xscaju-52","obj_delete_at_hint.0000000110","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-10:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.10:8","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-10:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.10:7","data.full-sync.index.e4b3955d-973b-4dd2-ae54-b7e681fbeb21.0","obj_delete_at_hint.0000000096","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.18-xscaju-18","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-11:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.11:8","obj_delete_at_hint.0000000087","obj_delete_at_hint.0000000008","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-16:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.16:1","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-19:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.19:8","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-12:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.12:7","obj_delete_at_hint.0000000006","obj_delete_at_hint.0000000029","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-11:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.11:7","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5174.2-xscaju-45","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.11-xscaju-11","obj_delete_at_hint.0000000089","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-20:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.20:8","obj_delete_at_hint.0000000022","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.6-xscaju-55","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-37:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.37:10","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-46:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5174.3:8","bilog.trim","obj_delete_at_hint.0000000016","obj_delete_at_hint.0000000041","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-17:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.17:3","data_log.0.ynNRd4MFfB1.0","obj_delete_at_hint.0000000018","obj_delete_at_hint.0000000092","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-22:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.22:8","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.6-xscaju-6","data_log.2.UQtue5bClUD.0","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-25:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.25:8","obj_delete_at_hint.0000000014","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-17:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.17:5","obj_delete_at_hint.0000000112","obj_delete_at_hint.0000000007","obj_delete_at_hint.0000000021","meta.log.7c4a812f-81e4-4a72-acce-0ee18589648c.0","obj_delete_at_hint.0000000064","obj_delete_at_hint.0000000071","obj_delete_at_hint.0000000074","datalog.sync-status.shard.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.2","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5375.1-xscaju-49","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-47:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5174.4:8","obj_delete_at_hint.0000000081","obj_delete_at_hint.0000000009","obj_delete_at_hint.0000000121","data_log.1","obj_delete_at_hint.0000000125","obj_delete_at_hint.0000000082","meta.full-sync.index.1","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.36-xscaju-36","obj_delete_at_hint.0000000105","obj_delete_at_hint.0000000059","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-21:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.21:8","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-22:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.22:8","obj_delete_at_hint.0000000077","obj_delete_at_hint.0000000032","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.7-xscaju-7","obj_delete_at_hint.0000000053","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-12:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.12:1","obj_delete_at_hint.0000000091","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.13-xscaju-13","obj_delete_at_hint.0000000065","obj_delete_at_hint.0000000083","obj_delete_at_hint.0000000010","data.full-sync.index.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.3","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-11:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.11:1","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5174.4-xscaju-47","datalog.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-37:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.37:10","mdlog.sync-status.shard.2","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-16:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.16:3","meta.log.7c4a812f-81e4-4a72-acce-0ee18589648c.1","obj_delete_at_hint.0000000045","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-23:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.23:8","obj_delete_at_hint.0000000002","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.24-xscaju-24","datalog.sync-status.shard.e4b3955d-973b-4dd2-ae54-b7e681fbeb21.0","obj_delete_at_hint.0000000116","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-11:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.11:3","obj_delete_at_hint.0000000034","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-10:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.10:1","obj_delete_at_hint.0000000101","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.39-xscaju-39","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.29-xscaju-29","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.25-xscaju-25","mdlog.sync-status.shard.1","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-12:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.12:1","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5174.3-xscaju-46","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-13:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.13:1","obj_delete_at_hint.0000000079","obj_delete_at_hint.0000000049","obj_delete_at_hint.0000000117","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-47:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5174.4:8","obj_delete_at_hint.0000000044","data.full-sync.index.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.2","obj_delete_at_hint.0000000066","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.32-xscaju-32","obj_delete_at_hint.0000000068","obj_delete_at_hint.0000000085","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-38:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.38:10","obj_delete_at_hint.0000000073","obj_delete_at_hint.0000000038","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.21-xscaju-21","meta.full-sync.index.3","obj_delete_at_hint.0000000118","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-23:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.23:8","obj_delete_at_hint.0000000036","datalog.sync-status.shard.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.3","obj_delete_at_hint.0000000103","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.3-xscaju-3","mdlog.sync-status.shard.3","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-49:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5375.1:5","obj_delete_at_hint.0000000119","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.22-xscaju-22","obj_delete_at_hint.0000000098","meta.log.2afe6ddd-1cf8-433a-ae51-67de25931f7b.0","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.38-xscaju-38","mdlog.sync-status.shard.0","2021-02-11-06-e4b3955d-973b-4dd2-ae54-b7e681fbeb21.5034.2-xscaju-43","obj_delete_at_hint.0000000027","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-24:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.24:8","mdlog.sync-status","obj_delete_at_hint.0000000019","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-19:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.19:5","obj_delete_at_hint.0000000039","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.10-xscaju-10","obj_delete_at_hint.0000000100","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-11:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.11:8","datalog.sync-status.shard.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.1","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-10:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.10:8","obj_delete_at_hint.0000000093","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-48:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5174.5:8","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.15-xscaju-15","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-12:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.12:7","obj_delete_at_hint.0000000004","obj_delete_at_hint.0000000063","obj_delete_at_hint.0000000122","obj_delete_at_hint.0000000057","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-15:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.15:1","obj_delete_at_hint.0000000054","obj_delete_at_hint.0000000114","data_log.0","obj_delete_at_hint.0000000012","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-38:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.38:10","obj_delete_at_hint.0000000084","obj_delete_at_hint.0000000043","obj_delete_at_hint.0000000111","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-14:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.14:1","obj_delete_at_hint.0000000048","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-19:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.19:8","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.8-xscaju-8","obj_delete_at_hint.0000000020","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.2-xscaju-2","obj_delete_at_hint.0000000099","obj_delete_at_hint.0000000056","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-10:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.10:7","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-18:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.18:3","obj_delete_at_hint.0000000072","obj_delete_at_hint.0000000062","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-18:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.18:1","obj_delete_at_hint.0000000115","obj_delete_at_hint.0000000033","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-16:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.16:5","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-24:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.24:8","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-18:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.18:5","datalog.sync-status.shard.e4b3955d-973b-4dd2-ae54-b7e681fbeb21.1","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-16:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.16:3","obj_delete_at_hint.0000000025","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-19:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.19:5","obj_delete_at_hint.0000000123","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.9-xscaju-9","obj_delete_at_hint.0000000108","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-15:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.15:1","obj_delete_at_hint.0000000094","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.20-xscaju-20","obj_delete_at_hint.0000000060","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-16:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.16:1","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-18:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.18:5","obj_delete_at_hint.0000000109","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.19-xscaju-19","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.40-xscaju-40","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-21:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.21:8","datalog.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21","obj_delete_at_hint.0000000013","data.full-sync.index.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.1","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.14-xscaju-14","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-17:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.17:1","obj_delete_at_hint.0000000051","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-48:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5174.5:8","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-11:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.11:1","obj_delete_at_hint.0000000106","obj_delete_at_hint.0000000017","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-12:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.12:8","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-36:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.36:10","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.12-xscaju-12","datalog.sync-status.shard.e4b3955d-973b-4dd2-ae54-b7e681fbeb21.2","obj_delete_at_hint.0000000023","obj_delete_at_hint.0000000107","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-46:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5174.3:8","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-10:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.10:1","obj_delete_at_hint.0000000046","meta.full-sync.index.2","datalog.sync-status.shard.e4b3955d-973b-4dd2-ae54-b7e681fbeb21.3","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-18:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.18:3","data_log.2","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-26:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.26:8","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-14:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.14:1","obj_delete_at_hint.0000000067","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-17:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.17:3","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-17:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.17:1","data.full-sync.index.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.0","obj_delete_at_hint.0000000086","data.full-sync.index.e4b3955d-973b-4dd2-ae54-b7e681fbeb21.2","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-13:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.13:1","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-18:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.18:1","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-20:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.20:8","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.9-xscaju-58","bucket.sync-status.e4b3955d-973b-4dd2-ae54-b7e681fbeb21:xscaju-11:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.11:7","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5174.5-xscaju-48","data_log.1.vOt3nMUokOr.0","obj_delete_at_hint.0000000050","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.5-xscaju-5","obj_delete_at_hint.0000000102","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-12:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.12:3","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5174.1-xscaju-44","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.41-xscaju-41","obj_delete_at_hint.0000000124","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.26-xscaju-26","obj_delete_at_hint.0000000001","obj_delete_at_hint.0000000126","obj_delete_at_hint.0000000055","obj_delete_at_hint.0000000061","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-36:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.36:10","obj_delete_at_hint.0000000097","obj_delete_at_hint.0000000042","obj_delete_at_hint.0000000075","obj_delete_at_hint.0000000120","obj_delete_at_hint.0000000030","2021-02-11-06-5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.23-xscaju-23"] 2021-02-11T06:36:46.141 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:36:46.141 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:36:46.141 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone4 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:36:46.178 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:36:46.178 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:36:46.415 INFO:teuthology.orchestra.run.smithi172.stdout:["obj_delete_at_hint.0000000078","meta.history","obj_delete_at_hint.0000000070","obj_delete_at_hint.0000000104","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-17:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.17:5","obj_delete_at_hint.0000000026","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-26:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.26:8","obj_delete_at_hint.0000000028","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-10:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.10:3","obj_delete_at_hint.0000000040","meta.full-sync.index.0","obj_delete_at_hint.0000000015","obj_delete_at_hint.0000000069","obj_delete_at_hint.0000000095","obj_delete_at_hint.0000000003","obj_delete_at_hint.0000000047","obj_delete_at_hint.0000000035","obj_delete_at_hint.0000000037","obj_delete_at_hint.0000000024","obj_delete_at_hint.0000000000","obj_delete_at_hint.0000000031","data_log.3","obj_delete_at_hint.0000000076","obj_delete_at_hint.0000000113","obj_delete_at_hint.0000000005","obj_delete_at_hint.0000000011","obj_delete_at_hint.0000000058","obj_delete_at_hint.0000000052","datalog.sync-status.shard.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.0","meta.log.0d061ff0-ede8-4baa-beef-d975d317d505.0","obj_delete_at_hint.0000000088","obj_delete_at_hint.0000000080","obj_delete_at_hint.0000000090","obj_delete_at_hint.0000000110","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-10:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.10:8","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-10:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.10:7","obj_delete_at_hint.0000000096","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-11:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.11:8","obj_delete_at_hint.0000000087","obj_delete_at_hint.0000000008","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-16:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.16:1","data_log.1.ky7Bbn68yhb.0","obj_delete_at_hint.0000000006","obj_delete_at_hint.0000000029","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-11:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.11:7","obj_delete_at_hint.0000000089","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-20:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.20:8","obj_delete_at_hint.0000000022","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-46:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5174.3:8","bilog.trim","obj_delete_at_hint.0000000016","obj_delete_at_hint.0000000041","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-17:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.17:3","obj_delete_at_hint.0000000018","obj_delete_at_hint.0000000092","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-22:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.22:8","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-25:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.25:8","obj_delete_at_hint.0000000014","obj_delete_at_hint.0000000112","obj_delete_at_hint.0000000007","obj_delete_at_hint.0000000021","meta.log.7c4a812f-81e4-4a72-acce-0ee18589648c.0","obj_delete_at_hint.0000000064","obj_delete_at_hint.0000000071","obj_delete_at_hint.0000000074","datalog.sync-status.shard.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.2","obj_delete_at_hint.0000000081","obj_delete_at_hint.0000000009","obj_delete_at_hint.0000000121","data_log.1","obj_delete_at_hint.0000000125","obj_delete_at_hint.0000000082","meta.full-sync.index.1","obj_delete_at_hint.0000000105","obj_delete_at_hint.0000000059","obj_delete_at_hint.0000000077","obj_delete_at_hint.0000000032","obj_delete_at_hint.0000000053","obj_delete_at_hint.0000000091","obj_delete_at_hint.0000000065","obj_delete_at_hint.0000000083","obj_delete_at_hint.0000000010","data.full-sync.index.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.3","datalog.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-37:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.37:10","mdlog.sync-status.shard.2","meta.log.7c4a812f-81e4-4a72-acce-0ee18589648c.1","obj_delete_at_hint.0000000045","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-23:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.23:8","obj_delete_at_hint.0000000002","obj_delete_at_hint.0000000116","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-11:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.11:3","obj_delete_at_hint.0000000034","obj_delete_at_hint.0000000101","mdlog.sync-status.shard.1","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-12:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.12:1","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-13:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.13:1","obj_delete_at_hint.0000000079","obj_delete_at_hint.0000000049","obj_delete_at_hint.0000000117","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-47:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5174.4:8","obj_delete_at_hint.0000000044","data.full-sync.index.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.2","obj_delete_at_hint.0000000066","obj_delete_at_hint.0000000068","obj_delete_at_hint.0000000085","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-38:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.38:10","obj_delete_at_hint.0000000073","obj_delete_at_hint.0000000038","meta.full-sync.index.3","obj_delete_at_hint.0000000118","obj_delete_at_hint.0000000036","datalog.sync-status.shard.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.3","obj_delete_at_hint.0000000103","mdlog.sync-status.shard.3","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-49:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5375.1:5","obj_delete_at_hint.0000000119","obj_delete_at_hint.0000000098","meta.log.2afe6ddd-1cf8-433a-ae51-67de25931f7b.0","mdlog.sync-status.shard.0","obj_delete_at_hint.0000000027","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-24:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.24:8","mdlog.sync-status","obj_delete_at_hint.0000000019","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-19:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.19:5","obj_delete_at_hint.0000000039","data_log.0.kV8SRgDCxt0.0","obj_delete_at_hint.0000000100","datalog.sync-status.shard.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.1","obj_delete_at_hint.0000000093","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-48:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5174.5:8","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-12:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.12:7","obj_delete_at_hint.0000000004","obj_delete_at_hint.0000000063","obj_delete_at_hint.0000000122","obj_delete_at_hint.0000000057","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-15:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.15:1","obj_delete_at_hint.0000000054","obj_delete_at_hint.0000000114","data_log.0","obj_delete_at_hint.0000000012","obj_delete_at_hint.0000000084","obj_delete_at_hint.0000000043","data_log.2.EBecgzU7ETi.0","obj_delete_at_hint.0000000111","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-14:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.14:1","obj_delete_at_hint.0000000048","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-19:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.19:8","data_log.3.OKoLALaz6VT.0","obj_delete_at_hint.0000000020","obj_delete_at_hint.0000000099","obj_delete_at_hint.0000000056","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-18:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.18:3","obj_delete_at_hint.0000000072","obj_delete_at_hint.0000000062","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-18:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.18:1","obj_delete_at_hint.0000000115","obj_delete_at_hint.0000000033","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-16:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.16:5","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-18:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.18:5","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-16:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.16:3","obj_delete_at_hint.0000000025","obj_delete_at_hint.0000000123","obj_delete_at_hint.0000000108","obj_delete_at_hint.0000000094","obj_delete_at_hint.0000000060","obj_delete_at_hint.0000000109","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-21:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.21:8","obj_delete_at_hint.0000000013","data.full-sync.index.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.1","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-17:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.17:1","obj_delete_at_hint.0000000051","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-11:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.11:1","obj_delete_at_hint.0000000106","obj_delete_at_hint.0000000017","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-12:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.12:8","obj_delete_at_hint.0000000023","obj_delete_at_hint.0000000107","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-10:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.10:1","obj_delete_at_hint.0000000046","meta.full-sync.index.2","data_log.2","obj_delete_at_hint.0000000067","data.full-sync.index.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.0","obj_delete_at_hint.0000000086","obj_delete_at_hint.0000000050","obj_delete_at_hint.0000000102","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-12:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.12:3","obj_delete_at_hint.0000000124","obj_delete_at_hint.0000000001","obj_delete_at_hint.0000000126","obj_delete_at_hint.0000000055","obj_delete_at_hint.0000000061","bucket.sync-status.5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-36:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.4335.36:10","obj_delete_at_hint.0000000097","obj_delete_at_hint.0000000042","obj_delete_at_hint.0000000075","obj_delete_at_hint.0000000120","obj_delete_at_hint.0000000030"] 2021-02-11T06:36:46.445 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:36:46.445 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:36:46.445 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:36:46.485 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:36:46.485 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:36:46.764 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"00000000000000000000:00000000000000007408","last_update":"2021-02-11T06:36:45.069979Z"},{"marker":"00000000000000000000:00000000000000003163","last_update":"2021-02-11T06:36:45.072954Z"},{"marker":"00000000000000000000:00000000000000004092","last_update":"2021-02-11T06:36:45.076221Z"},{"marker":"00000000000000000000:00000000000000005945","last_update":"2021-02-11T06:36:45.079376Z"}] 2021-02-11T06:36:46.765 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-02-11T06:36:46.765 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:36:46.765 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:36:46.765 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:36:46.806 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:36:46.807 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:36:47.096 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":15153533235123629405},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005032","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:32:32.945606Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000780","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:25:50.534339Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002507","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:21:37.634942Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003569","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:29:17.877002Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T06:36:47.097 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000007408 target=00000000000000000000:00000000000000005032, shard=1 master=00000000000000000000:00000000000000003163 target=00000000000000000000:00000000000000000780, shard=2 master=00000000000000000000:00000000000000004092 target=00000000000000000000:00000000000000002507, shard=3 master=00000000000000000000:00000000000000005945 target=00000000000000000000:00000000000000003569 2021-02-11T06:36:52.097 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:37:05.779 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:37:05.779 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:37:05.822 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:37:05.822 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:37:06.085 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":15153533235123629405},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000007408","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:36:45.069979Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003163","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:36:45.072954Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004092","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:36:45.076221Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005945","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:36:45.079376Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T06:37:06.086 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-02-11T06:37:06.086 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:37:06.086 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:37:06.086 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:37:06.125 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:37:06.126 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:37:06.411 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"00000000000000000000:00000000000000007408","last_update":"2021-02-11T06:36:45.069979Z"},{"marker":"00000000000000000000:00000000000000003163","last_update":"2021-02-11T06:36:45.072954Z"},{"marker":"00000000000000000000:00000000000000004092","last_update":"2021-02-11T06:36:45.076221Z"},{"marker":"00000000000000000000:00000000000000005945","last_update":"2021-02-11T06:36:45.079376Z"}] 2021-02-11T06:37:06.411 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-02-11T06:37:06.411 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:37:06.411 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:37:06.412 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:37:06.495 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:37:06.495 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:37:06.765 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":15040254564068855740},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000007408","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:36:45.069979Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003163","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:36:45.072954Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004092","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:36:45.076221Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005945","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:36:45.079376Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T06:37:06.766 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-02-11T06:37:06.766 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:37:06.766 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:37:06.766 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:37:06.846 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:37:06.846 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:37:07.123 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"00000000000000000000:00000000000000007408","last_update":"2021-02-11T06:36:45.069979Z"},{"marker":"00000000000000000000:00000000000000003163","last_update":"2021-02-11T06:36:45.072954Z"},{"marker":"00000000000000000000:00000000000000004092","last_update":"2021-02-11T06:36:45.076221Z"},{"marker":"00000000000000000000:00000000000000005945","last_update":"2021-02-11T06:36:45.079376Z"}] 2021-02-11T06:37:07.123 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-02-11T06:37:07.124 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:37:07.124 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:37:07.124 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:37:07.167 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:37:07.167 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:37:07.418 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":1899655024930128290},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000007408","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:36:45.069979Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003163","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:36:45.072954Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004092","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:36:45.076221Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005945","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:36:45.079376Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T06:37:07.419 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-02-11T06:37:07.419 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:37:07.419 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:37:07.419 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:37:07.500 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:37:07.500 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:37:07.777 INFO:teuthology.orchestra.run.smithi172.stdout:[{"marker":"00000000000000000000:00000000000000006220","last_update":"2021-02-11T06:36:45.572017Z"},{"marker":"00000000000000000000:00000000000000001972","last_update":"2021-02-11T06:36:45.576513Z"},{"marker":"00000000000000000000:00000000000000003167","last_update":"2021-02-11T06:36:45.565334Z"},{"marker":"00000000000000000000:00000000000000004757","last_update":"2021-02-11T06:36:45.567996Z"}] 2021-02-11T06:37:07.777 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-02-11T06:37:07.777 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:37:07.777 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:37:07.778 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:37:07.821 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:37:07.821 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:37:08.101 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":3699056310506273018},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000006220","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:36:45.572017Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000001972","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:36:45.576513Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003167","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:36:45.565334Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004757","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:36:45.567996Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T06:37:08.101 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-02-11T06:37:08.101 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:37:08.101 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:37:08.102 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:37:08.142 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:37:08.143 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:37:08.405 INFO:teuthology.orchestra.run.smithi172.stdout:[{"marker":"00000000000000000000:00000000000000006220","last_update":"2021-02-11T06:36:45.572017Z"},{"marker":"00000000000000000000:00000000000000001972","last_update":"2021-02-11T06:36:45.576513Z"},{"marker":"00000000000000000000:00000000000000003167","last_update":"2021-02-11T06:36:45.565334Z"},{"marker":"00000000000000000000:00000000000000004757","last_update":"2021-02-11T06:36:45.567996Z"}] 2021-02-11T06:37:19.494 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-02-11T06:37:19.495 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:37:19.495 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:37:19.495 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:37:19.539 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:37:19.540 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:37:19.818 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":995053126825359944},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000006220","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:36:45.572017Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000001972","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:36:45.576513Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003167","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:36:45.565334Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004757","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:36:45.567996Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T06:37:19.818 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-02-11T06:37:19.819 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:37:19.819 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:37:19.819 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:37:19.900 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:37:19.901 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:37:20.186 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"00000000000000000000:00000000000000006088","last_update":"2021-02-11T06:36:45.563961Z"},{"marker":"00000000000000000000:00000000000000001972","last_update":"2021-02-11T06:36:45.566070Z"},{"marker":"00000000000000000000:00000000000000003167","last_update":"2021-02-11T06:36:45.559330Z"},{"marker":"00000000000000000000:00000000000000004757","last_update":"2021-02-11T06:36:45.561946Z"}] 2021-02-11T06:37:20.186 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-02-11T06:37:20.186 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:37:20.186 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:37:20.187 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:37:20.273 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:37:20.274 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:37:20.545 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":398085786125260623},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000006088","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:36:45.563961Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000001972","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:36:45.566070Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003167","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:36:45.559330Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004757","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:36:45.561946Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T06:37:20.546 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-02-11T06:37:20.546 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:37:20.546 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:37:20.547 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:37:20.630 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:37:20.630 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:37:20.909 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"00000000000000000000:00000000000000006088","last_update":"2021-02-11T06:36:45.563961Z"},{"marker":"00000000000000000000:00000000000000001972","last_update":"2021-02-11T06:36:45.566070Z"},{"marker":"00000000000000000000:00000000000000003167","last_update":"2021-02-11T06:36:45.559330Z"},{"marker":"00000000000000000000:00000000000000004757","last_update":"2021-02-11T06:36:45.561946Z"}] 2021-02-11T06:37:20.910 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-02-11T06:37:20.910 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:37:20.910 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:37:20.910 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:37:20.953 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:37:20.954 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:37:21.226 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":11837782300501369896},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000006088","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:36:45.563961Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000001972","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:36:45.566070Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003167","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:36:45.559330Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004757","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:36:45.561946Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T06:37:21.227 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-02-11T06:37:21.227 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_sync_disable ... ok 2021-02-11T06:37:21.228 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xscaju-59 2021-02-11T06:37:21.271 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xscaju-60 2021-02-11T06:37:21.350 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xscaju-61 2021-02-11T06:37:22.028 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:37:22.028 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:37:22.028 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:37:22.076 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:37:22.077 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:37:22.292 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=0d061ff0-ede8-4baa-beef-d975d317d505 2021-02-11T06:37:22.341 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613025441.394344_242.1","last_update":"2021-02-11T06:37:21.394344Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:37:22.341 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T06:37:22.341 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:37:22.341 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:37:22.342 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:37:22.380 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:37:22.381 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:37:22.628 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613025441.394344_242.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:37:21.394344Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:37:22.629 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T06:37:22.629 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T06:37:22.629 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:37:22.629 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:37:22.629 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:37:22.672 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:37:22.672 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:37:22.958 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613025441.394344_242.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:37:21.394344Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:37:22.958 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T06:37:22.958 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:37:22.959 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:37:22.959 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:37:22.959 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:37:22.999 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:37:22.999 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:37:23.257 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613025441.394344_242.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:37:21.394344Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:37:23.258 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:37:23.258 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:37:23.258 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:37:23.259 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-59 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:37:23.350 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:37:23.351 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:37:23.602 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:37:23.596+0000 7f381997c080 1 bucket sync caught up with source: 2021-02-11T06:37:23.603 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [00000000001.24.5, , 00000000001.51.5, , 00000000001.54.5, , , 00000000001.44.5, , , ] 2021-02-11T06:37:23.603 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [00000000001.24.5, , 00000000001.51.5, , 00000000001.54.5, , , 00000000001.44.5, , , ] 2021-02-11T06:37:23.603 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:37:23.596+0000 7f381997c080 0 bucket checkpoint complete 2021-02-11T06:37:23.635 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:37:23.636 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:37:23.636 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-59 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:37:23.690 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:37:23.691 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:37:23.952 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:37:23.947+0000 7f802542a080 1 bucket sync caught up with source: 2021-02-11T06:37:23.952 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [00000000001.24.5, , 00000000001.51.5, , 00000000001.54.5, , , 00000000001.44.5, , , ] 2021-02-11T06:37:23.952 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [00000000001.24.5, , 00000000001.51.5, , 00000000001.54.5, , , 00000000001.44.5, , , ] 2021-02-11T06:37:23.952 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:37:23.947+0000 7f802542a080 0 bucket checkpoint complete 2021-02-11T06:37:23.979 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:37:23.980 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:37:23.980 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-59 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:37:24.028 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:37:24.029 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:37:24.261 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:37:24.252+0000 7f179fd9f080 1 bucket sync caught up with source: 2021-02-11T06:37:24.262 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [00000000001.24.5, , 00000000001.51.5, , 00000000001.54.5, , , 00000000001.44.5, , , ] 2021-02-11T06:37:24.262 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [00000000001.24.5, , 00000000001.51.5, , 00000000001.54.5, , , 00000000001.44.5, , , ] 2021-02-11T06:37:24.262 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:37:24.252+0000 7f179fd9f080 0 bucket checkpoint complete 2021-02-11T06:37:24.286 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-59', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:37:24.286 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-59', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:37:24.287 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-59 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:37:24.331 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:37:24.332 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:37:24.583 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:37:24.579+0000 7f532586d080 1 bucket sync caught up with source: 2021-02-11T06:37:24.584 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [00000000001.27.5, , 00000000001.51.5, , 00000000001.59.5, , , 00000000001.57.5, , , ] 2021-02-11T06:37:24.584 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [00000000001.27.5, , 00000000001.51.5, , 00000000001.59.5, , , 00000000001.57.5, , , ] 2021-02-11T06:37:24.584 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:37:24.579+0000 7f532586d080 0 bucket checkpoint complete 2021-02-11T06:37:24.616 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-59', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:37:24.617 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-59', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:37:24.617 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-59 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:37:24.709 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:37:24.710 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:37:24.978 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:37:24.975+0000 7f2fea1ef080 1 bucket sync caught up with source: 2021-02-11T06:37:24.978 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [00000000001.27.5, , 00000000001.51.5, , 00000000001.59.5, , , 00000000001.57.5, , , ] 2021-02-11T06:37:24.978 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [00000000001.27.5, , 00000000001.51.5, , 00000000001.59.5, , , 00000000001.57.5, , , ] 2021-02-11T06:37:24.978 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:37:24.975+0000 7f2fea1ef080 0 bucket checkpoint complete 2021-02-11T06:37:25.010 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-59', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:37:25.010 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-59', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:37:25.011 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-59 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:37:25.056 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:37:25.057 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:37:25.317 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:37:25.315+0000 7f18aeb30080 1 bucket sync caught up with source: 2021-02-11T06:37:25.318 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [00000000001.27.5, , 00000000001.58.5, , 00000000001.57.5, , , 00000000001.51.5, , , ] 2021-02-11T06:37:25.318 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [00000000001.27.5, , 00000000001.58.5, , 00000000001.57.5, , , 00000000001.51.5, , , ] 2021-02-11T06:37:25.318 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:37:25.315+0000 7f18aeb30080 0 bucket checkpoint complete 2021-02-11T06:37:25.354 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-59', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:37:25.354 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-59', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:37:25.354 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-59 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:37:25.402 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:37:25.402 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:37:25.643 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:37:25.636+0000 7f8832485080 1 bucket sync caught up with source: 2021-02-11T06:37:25.643 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [00000000001.27.5, , 00000000001.58.5, , 00000000001.57.5, , , 00000000001.51.5, , , ] 2021-02-11T06:37:25.643 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [00000000001.27.5, , 00000000001.58.5, , 00000000001.57.5, , , 00000000001.51.5, , , ] 2021-02-11T06:37:25.643 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:37:25.636+0000 7f8832485080 0 bucket checkpoint complete 2021-02-11T06:37:25.675 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-59 zones={test-zone2, test-zone1} 2021-02-11T06:37:25.685 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:37:25.691 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T06:37:25.691 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T06:37:25.691 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T06:37:25.691 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T06:37:25.691 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:37:25.698 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T06:37:25.698 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T06:37:25.698 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T06:37:25.698 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T06:37:25.711 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:37:25.764 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:37:25.879 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:37:25.991 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:37:26.048 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:37:26.172 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:37:26.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:37:26.336 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:37:26.453 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:37:26.568 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:37:26.629 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:37:26.744 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:37:26.855 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-59 zones={test-zone2, test-zone1} 2021-02-11T06:37:26.855 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-59 zones={test-zone3, test-zone1} 2021-02-11T06:37:26.864 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:37:26.869 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T06:37:26.869 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T06:37:26.869 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T06:37:26.869 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T06:37:26.869 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:37:26.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T06:37:26.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T06:37:26.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T06:37:26.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T06:37:26.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:37:26.939 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:37:27.056 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:37:27.167 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:37:27.224 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:37:31.167 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:37:31.279 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:37:31.336 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:37:31.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:37:31.552 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:37:31.612 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:37:31.727 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:37:31.830 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-59 zones={test-zone3, test-zone1} 2021-02-11T06:37:31.831 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-59 zones={test-zone3, test-zone2} 2021-02-11T06:37:31.843 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:37:31.849 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T06:37:31.849 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T06:37:31.849 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T06:37:31.849 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T06:37:31.849 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:37:31.855 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T06:37:31.855 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T06:37:31.855 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T06:37:31.855 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T06:37:31.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:37:31.924 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:37:32.041 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:37:32.154 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:37:32.208 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:37:32.326 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:37:32.446 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:37:32.515 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:37:32.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:37:32.760 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:37:32.816 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:37:32.932 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:37:47.158 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-59 zones={test-zone3, test-zone2} 2021-02-11T06:37:47.158 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:37:47.159 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:37:47.159 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-60 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:37:47.203 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:37:47.203 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:37:47.453 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:37:47.443+0000 7febea370080 1 bucket sync caught up with source: 2021-02-11T06:37:47.453 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [00000000001.54.5, , 00000000001.36.5, , 00000000001.59.5, , , 00000000001.49.5, , , ] 2021-02-11T06:37:47.453 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [00000000001.54.5, , 00000000001.36.5, , 00000000001.59.5, , , 00000000001.49.5, , , ] 2021-02-11T06:37:47.454 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:37:47.443+0000 7febea370080 0 bucket checkpoint complete 2021-02-11T06:37:47.486 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:37:47.486 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:37:47.487 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-60 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:37:47.535 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:37:47.535 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:37:47.791 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:37:47.787+0000 7f98df4c4080 1 bucket sync caught up with source: 2021-02-11T06:38:00.102 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [00000000001.54.5, , 00000000001.36.5, , 00000000001.59.5, , , 00000000001.49.5, , , ] 2021-02-11T06:38:00.103 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [00000000001.54.5, , 00000000001.36.5, , 00000000001.59.5, , , 00000000001.49.5, , , ] 2021-02-11T06:38:00.103 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:37:47.787+0000 7f98df4c4080 0 bucket checkpoint complete 2021-02-11T06:38:00.105 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:38:00.106 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:38:00.106 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-60 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:38:00.153 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:38:00.153 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:38:00.392 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:38:00.387+0000 7fe9dc26e080 1 bucket sync caught up with source: 2021-02-11T06:38:00.392 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [00000000001.54.5, , 00000000001.36.5, , 00000000001.59.5, , , 00000000001.49.5, , , ] 2021-02-11T06:38:00.392 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [00000000001.54.5, , 00000000001.36.5, , 00000000001.59.5, , , 00000000001.49.5, , , ] 2021-02-11T06:38:00.392 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:38:00.387+0000 7fe9dc26e080 0 bucket checkpoint complete 2021-02-11T06:38:00.424 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-60', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:38:00.424 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-60', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:38:00.424 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-60 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:38:00.473 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:38:00.473 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:38:00.730 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:38:00.727+0000 7f6191754080 1 bucket sync caught up with source: 2021-02-11T06:38:00.857 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [00000000001.78.5, , 00000000001.40.5, , 00000000001.65.5, , , 00000000001.51.5, , , ] 2021-02-11T06:38:00.857 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [00000000001.78.5, , 00000000001.40.5, , 00000000001.65.5, , , 00000000001.51.5, , , ] 2021-02-11T06:38:00.857 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:38:00.727+0000 7f6191754080 0 bucket checkpoint complete 2021-02-11T06:38:00.859 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-60', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:38:00.859 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-60', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:38:00.859 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-60 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:38:00.908 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:38:00.908 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:38:01.173 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:38:01.171+0000 7f653fa4c080 1 bucket sync caught up with source: 2021-02-11T06:38:01.173 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [00000000001.78.5, , 00000000001.40.5, , 00000000001.65.5, , , 00000000001.51.5, , , ] 2021-02-11T06:38:01.173 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [00000000001.78.5, , 00000000001.40.5, , 00000000001.65.5, , , 00000000001.51.5, , , ] 2021-02-11T06:38:01.173 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:38:01.171+0000 7f653fa4c080 0 bucket checkpoint complete 2021-02-11T06:38:01.196 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-60', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:38:01.196 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-60', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:38:01.197 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-60 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:38:01.242 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:38:01.243 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:38:01.487 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:38:01.483+0000 7f4074dcc080 1 bucket sync caught up with source: 2021-02-11T06:38:01.487 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [00000000001.51.5, , 00000000001.39.5, , 00000000001.66.5, , , 00000000001.46.5, , , ] 2021-02-11T06:38:01.487 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [00000000001.51.5, , 00000000001.39.5, , 00000000001.66.5, , , 00000000001.46.5, , , ] 2021-02-11T06:38:01.487 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:38:01.483+0000 7f4074dcc080 0 bucket checkpoint complete 2021-02-11T06:38:01.515 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-60', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:38:01.515 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-60', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:38:01.515 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-60 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:38:01.567 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:38:01.567 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:38:01.821 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:38:01.815+0000 7f91e4156080 1 bucket sync caught up with source: 2021-02-11T06:38:01.821 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [00000000001.51.5, , 00000000001.39.5, , 00000000001.66.5, , , 00000000001.46.5, , , ] 2021-02-11T06:38:01.821 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [00000000001.51.5, , 00000000001.39.5, , 00000000001.66.5, , , 00000000001.46.5, , , ] 2021-02-11T06:38:01.821 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:38:01.815+0000 7f91e4156080 0 bucket checkpoint complete 2021-02-11T06:38:01.857 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-60 zones={test-zone2, test-zone1} 2021-02-11T06:38:01.868 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:38:01.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T06:38:01.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T06:38:01.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T06:38:01.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T06:38:01.874 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:38:01.880 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T06:38:01.880 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T06:38:01.880 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T06:38:01.880 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T06:38:01.893 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:38:01.952 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:38:02.065 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:38:02.171 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:38:02.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:38:02.335 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:38:02.451 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:38:02.508 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:38:02.620 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:38:02.735 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:38:02.788 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:38:02.919 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:38:03.542 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-60 zones={test-zone2, test-zone1} 2021-02-11T06:38:03.542 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-60 zones={test-zone3, test-zone1} 2021-02-11T06:38:03.552 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:38:03.595 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T06:38:03.595 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T06:38:03.595 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T06:38:03.595 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T06:38:03.595 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:38:03.600 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T06:38:03.601 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T06:38:03.601 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T06:38:03.601 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T06:38:03.662 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:38:03.716 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:38:03.831 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:38:03.979 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:38:04.038 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:38:04.146 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:38:04.250 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:38:04.304 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:38:04.428 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:38:04.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:38:04.588 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:38:04.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:38:04.810 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-60 zones={test-zone3, test-zone1} 2021-02-11T06:38:04.810 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-60 zones={test-zone3, test-zone2} 2021-02-11T06:38:04.821 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:38:04.826 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T06:38:04.826 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T06:38:04.826 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T06:38:04.826 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T06:38:04.826 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:38:04.832 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T06:38:04.832 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T06:38:04.832 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T06:38:04.832 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T06:38:04.886 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:38:04.944 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:38:05.060 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:38:05.166 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:38:05.220 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:38:05.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:38:05.439 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:38:05.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:38:05.620 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:38:05.730 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:38:05.784 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:38:05.896 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:38:06.002 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-60 zones={test-zone3, test-zone2} 2021-02-11T06:38:06.002 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:38:06.002 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:38:06.002 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-61 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:38:06.044 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:38:06.044 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:38:06.283 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:38:06.279+0000 7f072cdd1080 1 bucket sync caught up with source: 2021-02-11T06:38:06.283 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [00000000001.61.5, , 00000000001.32.5, , 00000000001.61.5, , , 00000000001.34.5, , , ] 2021-02-11T06:38:06.283 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [00000000001.61.5, , 00000000001.32.5, , 00000000001.61.5, , , 00000000001.34.5, , , ] 2021-02-11T06:38:06.283 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:38:06.279+0000 7f072cdd1080 0 bucket checkpoint complete 2021-02-11T06:38:06.313 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:38:06.313 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:38:06.313 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-61 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:38:06.362 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:38:06.362 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:38:06.621 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:38:06.619+0000 7f49312af080 1 bucket sync caught up with source: 2021-02-11T06:38:06.621 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [00000000001.61.5, , 00000000001.32.5, , 00000000001.61.5, , , 00000000001.34.5, , , ] 2021-02-11T06:38:06.621 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [00000000001.61.5, , 00000000001.32.5, , 00000000001.61.5, , , 00000000001.34.5, , , ] 2021-02-11T06:38:06.621 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:38:06.619+0000 7f49312af080 0 bucket checkpoint complete 2021-02-11T06:38:06.650 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:38:06.650 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:38:06.650 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-61 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:38:06.698 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:38:06.698 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:38:06.954 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:38:06.947+0000 7fd2e2240080 1 bucket sync caught up with source: 2021-02-11T06:38:06.954 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [00000000001.61.5, , 00000000001.32.5, , 00000000001.61.5, , , 00000000001.34.5, , , ] 2021-02-11T06:38:06.954 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [00000000001.61.5, , 00000000001.32.5, , 00000000001.61.5, , , 00000000001.34.5, , , ] 2021-02-11T06:38:06.954 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:38:06.947+0000 7fd2e2240080 0 bucket checkpoint complete 2021-02-11T06:38:06.983 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-61', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:38:06.984 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-61', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:38:06.984 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-61 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:38:07.030 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:38:07.031 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:38:07.294 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:38:07.291+0000 7f572037a080 1 bucket sync caught up with source: 2021-02-11T06:38:07.294 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [00000000001.67.5, , 00000000001.32.5, , 00000000001.67.5, , , 00000000001.34.5, , , ] 2021-02-11T06:38:07.295 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [00000000001.67.5, , 00000000001.32.5, , 00000000001.67.5, , , 00000000001.34.5, , , ] 2021-02-11T06:38:07.295 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:38:07.291+0000 7f572037a080 0 bucket checkpoint complete 2021-02-11T06:38:07.327 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-61', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:38:07.327 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-61', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:38:07.327 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-61 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:38:07.375 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:38:07.375 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:38:07.647 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:38:07.643+0000 7f8121d58080 1 bucket sync caught up with source: 2021-02-11T06:38:07.647 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [00000000001.67.5, , 00000000001.32.5, , 00000000001.67.5, , , 00000000001.34.5, , , ] 2021-02-11T06:38:07.647 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [00000000001.67.5, , 00000000001.32.5, , 00000000001.67.5, , , 00000000001.34.5, , , ] 2021-02-11T06:38:07.647 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:38:07.643+0000 7f8121d58080 0 bucket checkpoint complete 2021-02-11T06:38:07.680 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-61', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:38:07.680 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-61', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:38:07.680 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-61 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:38:07.728 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:38:07.728 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:38:07.988 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:38:07.983+0000 7f10fb7d8080 1 bucket sync caught up with source: 2021-02-11T06:38:07.988 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [00000000001.68.5, , 00000000001.30.5, , 00000000001.92.5, , , 00000000001.32.5, , , ] 2021-02-11T06:38:07.988 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [00000000001.68.5, , 00000000001.30.5, , 00000000001.92.5, , , 00000000001.32.5, , , ] 2021-02-11T06:38:07.988 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:38:07.983+0000 7f10fb7d8080 0 bucket checkpoint complete 2021-02-11T06:38:08.014 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-61', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:38:08.014 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-61', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:38:08.015 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-61 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:38:08.064 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:38:08.064 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:38:08.306 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:38:08.299+0000 7fa378d9f080 1 bucket sync caught up with source: 2021-02-11T06:38:08.306 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [00000000001.68.5, , 00000000001.30.5, , 00000000001.92.5, , , 00000000001.32.5, , , ] 2021-02-11T06:38:08.307 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [00000000001.68.5, , 00000000001.30.5, , 00000000001.92.5, , , 00000000001.32.5, , , ] 2021-02-11T06:38:08.307 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:38:08.299+0000 7fa378d9f080 0 bucket checkpoint complete 2021-02-11T06:38:08.338 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-61 zones={test-zone2, test-zone1} 2021-02-11T06:38:08.347 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:38:08.399 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T06:38:21.871 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T06:38:21.871 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T06:38:21.871 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T06:38:21.871 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:38:21.879 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T06:38:21.879 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T06:38:21.879 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T06:38:21.879 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T06:38:21.893 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:38:21.962 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:38:22.121 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:38:28.773 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:38:28.828 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:38:28.940 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:38:29.054 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:38:29.109 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:38:29.222 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:38:29.326 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:38:29.378 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:38:29.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:38:29.611 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-61 zones={test-zone2, test-zone1} 2021-02-11T06:38:29.611 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-61 zones={test-zone3, test-zone1} 2021-02-11T06:38:29.621 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:38:29.667 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T06:38:29.667 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T06:38:29.668 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T06:38:29.668 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T06:38:29.668 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:38:29.675 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T06:38:29.675 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T06:38:29.675 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T06:38:29.675 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T06:38:29.733 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:38:29.787 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:38:29.904 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:38:30.012 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:38:30.072 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:38:30.184 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:38:30.290 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:38:30.345 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:38:30.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:38:30.562 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:38:30.616 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:38:30.739 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:38:30.846 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-61 zones={test-zone3, test-zone1} 2021-02-11T06:38:30.846 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-61 zones={test-zone3, test-zone2} 2021-02-11T06:38:30.858 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:38:30.863 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T06:38:30.863 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T06:38:30.863 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T06:38:30.863 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T06:38:30.863 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:38:30.869 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T06:38:30.870 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T06:38:30.870 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T06:38:30.870 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T06:38:30.925 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:38:30.981 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:38:31.239 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:38:31.344 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:38:31.403 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:38:31.515 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:38:31.625 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:38:31.680 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:38:31.818 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:38:31.943 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:38:32.006 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:38:32.120 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:38:32.230 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-61 zones={test-zone3, test-zone2} 2021-02-11T06:38:32.231 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'xscaju-59', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:38:32.231 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'xscaju-59', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:38:32.231 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket xscaju-59 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T06:38:32.275 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:38:32.275 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:38:32.657 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'xscaju-59', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:38:32.657 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'xscaju-59', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:38:32.657 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket xscaju-59 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T06:38:32.740 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:38:32.740 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:38:33.110 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'xscaju-60', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:38:45.580 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'xscaju-60', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:38:45.580 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket xscaju-60 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T06:38:45.624 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:38:45.624 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:38:45.974 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'xscaju-60', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:38:45.974 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'xscaju-60', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:38:45.974 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket xscaju-60 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T06:38:46.016 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:38:46.016 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:38:46.377 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'xscaju-61', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:38:46.377 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'xscaju-61', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:38:46.378 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket xscaju-61 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T06:38:46.460 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:38:46.460 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:38:46.809 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'xscaju-61', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:38:46.809 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'xscaju-61', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:38:46.809 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket xscaju-61 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T06:38:46.893 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:38:46.894 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:38:47.876 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:38:51.466 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:38:51.467 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-59 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:38:51.515 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:38:51.516 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:38:51.773 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:38:51.766+0000 7f5bee25e080 1 bucket sync caught up with source: 2021-02-11T06:38:51.773 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [00000000006.30.5, 00000000004.60.1, 00000000006.65.5, 00000000004.42.1, 00000000006.60.5, 00000000004.53.1, 00000000004.62.1, 00000000006.50.5, 00000000004.62.1, 00000000004.37.1, 00000000004.63.1] 2021-02-11T06:38:51.774 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [00000000006.30.5, 00000000004.60.1, 00000000006.65.5, 00000000004.42.1, 00000000006.60.5, 00000000004.53.1, 00000000004.62.1, 00000000006.50.5, 00000000004.62.1, 00000000004.37.1, 00000000004.63.1] 2021-02-11T06:38:51.774 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:38:51.766+0000 7f5bee25e080 0 bucket checkpoint complete 2021-02-11T06:38:51.806 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:38:51.807 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:38:51.807 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-59 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:38:51.854 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:38:51.855 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:38:52.108 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:38:52.102+0000 7fe4e2392080 1 bucket sync caught up with source: 2021-02-11T06:38:52.108 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [00000000006.30.5, 00000000004.60.1, 00000000006.65.5, 00000000004.42.1, 00000000006.60.5, 00000000004.53.1, 00000000004.62.1, 00000000006.50.5, 00000000004.62.1, 00000000004.37.1, 00000000004.63.1] 2021-02-11T06:38:52.108 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [00000000006.30.5, 00000000004.60.1, 00000000006.65.5, 00000000004.42.1, 00000000006.60.5, 00000000004.53.1, 00000000004.62.1, 00000000006.50.5, 00000000004.62.1, 00000000004.37.1, 00000000004.63.1] 2021-02-11T06:38:52.109 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:38:52.102+0000 7fe4e2392080 0 bucket checkpoint complete 2021-02-11T06:38:52.135 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:38:52.135 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:38:52.135 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-59 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:38:52.184 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:38:52.185 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:38:52.422 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:38:52.414+0000 7f76315e6080 1 bucket sync caught up with source: 2021-02-11T06:38:52.422 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [00000000006.30.5, 00000000004.60.1, 00000000006.65.5, 00000000004.42.1, 00000000006.60.5, 00000000004.53.1, 00000000004.62.1, 00000000006.50.5, 00000000004.62.1, 00000000004.37.1, 00000000004.63.1] 2021-02-11T06:38:52.422 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [00000000006.30.5, 00000000004.60.1, 00000000006.65.5, 00000000004.42.1, 00000000006.60.5, 00000000004.53.1, 00000000004.62.1, 00000000006.50.5, 00000000004.62.1, 00000000004.37.1, 00000000004.63.1] 2021-02-11T06:38:52.423 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:38:52.414+0000 7f76315e6080 0 bucket checkpoint complete 2021-02-11T06:38:52.454 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-59', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:38:52.454 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-59', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:38:52.454 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-59 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:38:52.502 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:38:52.502 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:38:52.751 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:38:52.746+0000 7fe9a0092080 1 bucket sync caught up with source: 2021-02-11T06:38:52.751 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [00000000002.31.5, , 00000000002.59.5, , 00000000002.63.5, , , 00000000002.61.5, , , ] 2021-02-11T06:38:52.751 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [00000000002.31.5, , 00000000002.59.5, , 00000000002.63.5, , , 00000000002.61.5, , , ] 2021-02-11T06:38:52.751 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:38:52.746+0000 7fe9a0092080 0 bucket checkpoint complete 2021-02-11T06:38:52.784 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-59', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:38:52.784 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-59', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:38:52.785 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-59 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:38:52.833 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:38:52.833 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:38:53.103 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:38:53.098+0000 7f1d6be20080 1 bucket sync caught up with source: 2021-02-11T06:38:59.754 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [00000000002.31.5, , 00000000002.59.5, , 00000000002.63.5, , , 00000000002.61.5, , , ] 2021-02-11T06:38:59.755 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [00000000002.31.5, , 00000000002.59.5, , 00000000002.63.5, , , 00000000002.61.5, , , ] 2021-02-11T06:38:59.755 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:38:53.098+0000 7f1d6be20080 0 bucket checkpoint complete 2021-02-11T06:38:59.757 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-59', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:38:59.758 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-59', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:38:59.758 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-59 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:38:59.809 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:38:59.810 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:39:00.071 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:39:00.066+0000 7fc5b99a6080 1 bucket sync caught up with source: 2021-02-11T06:39:00.071 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [00000000002.31.5, , 00000000002.66.5, , 00000000002.61.5, , , 00000000002.55.5, , , ] 2021-02-11T06:39:00.071 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [00000000002.31.5, , 00000000002.66.5, , 00000000002.61.5, , , 00000000002.55.5, , , ] 2021-02-11T06:39:00.072 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:39:00.066+0000 7fc5b99a6080 0 bucket checkpoint complete 2021-02-11T06:39:00.096 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-59', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:39:00.097 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-59', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:39:00.097 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-59 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:39:00.145 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:39:00.145 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:39:00.377 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:39:00.369+0000 7f469ed54080 1 bucket sync caught up with source: 2021-02-11T06:39:00.378 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [00000000002.31.5, , 00000000002.66.5, , 00000000002.61.5, , , 00000000002.55.5, , , ] 2021-02-11T06:39:00.378 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [00000000002.31.5, , 00000000002.66.5, , 00000000002.61.5, , , 00000000002.55.5, , , ] 2021-02-11T06:39:00.378 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:39:00.369+0000 7f469ed54080 0 bucket checkpoint complete 2021-02-11T06:39:00.411 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-59 zones={test-zone2, test-zone1} 2021-02-11T06:39:00.423 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:39:00.430 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T06:39:00.430 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T06:39:00.431 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T06:39:00.431 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T06:39:00.431 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T06:39:00.431 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T06:39:00.431 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T06:39:00.431 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T06:39:00.431 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:39:00.438 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T06:39:00.438 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T06:39:00.438 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T06:39:00.438 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T06:39:00.438 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T06:39:00.439 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T06:39:00.439 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T06:39:00.439 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T06:39:00.453 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:39:00.510 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:39:00.629 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:39:00.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:39:00.801 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:39:00.916 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:39:01.032 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:39:01.088 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:39:01.202 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:39:01.308 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:39:01.358 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:39:01.469 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:39:01.579 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T06:39:01.636 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T06:39:01.753 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T06:39:01.859 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T06:39:01.916 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T06:39:02.034 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T06:39:02.141 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T06:39:02.196 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T06:39:02.312 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T06:39:02.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T06:39:02.489 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T06:39:02.638 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T06:39:06.895 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-59 zones={test-zone2, test-zone1} 2021-02-11T06:39:06.895 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-59 zones={test-zone3, test-zone1} 2021-02-11T06:39:06.905 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:39:06.910 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T06:39:06.911 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T06:39:06.911 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T06:39:06.911 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T06:39:06.911 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T06:39:06.911 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T06:39:06.911 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T06:39:06.911 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T06:39:06.911 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:39:06.917 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T06:39:06.917 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T06:39:06.917 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T06:39:06.917 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T06:39:06.917 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T06:39:06.917 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T06:39:06.917 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T06:39:06.917 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T06:39:06.928 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:39:06.981 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:39:07.092 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:39:07.199 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:39:07.255 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:39:07.369 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:39:07.476 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:39:07.532 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:39:07.650 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:39:07.759 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:39:07.812 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:39:07.924 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:39:08.035 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T06:39:08.096 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T06:39:08.208 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T06:39:08.318 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T06:39:08.371 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T06:39:08.488 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T06:39:08.594 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T06:39:08.658 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T06:39:37.406 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T06:39:37.515 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T06:39:37.568 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T06:39:37.708 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T06:39:40.634 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-59 zones={test-zone3, test-zone1} 2021-02-11T06:39:40.634 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-59 zones={test-zone3, test-zone2} 2021-02-11T06:39:40.646 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:39:40.651 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T06:39:40.652 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T06:39:40.652 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T06:39:40.652 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T06:39:40.652 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T06:39:40.652 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T06:39:40.652 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T06:39:40.652 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T06:39:40.652 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:39:40.658 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T06:39:40.658 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T06:39:40.658 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T06:39:40.658 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T06:39:40.659 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T06:39:40.659 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T06:39:40.659 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T06:39:40.659 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T06:39:40.671 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:39:40.724 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:39:40.837 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:39:40.946 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:39:41.000 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:39:41.113 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:39:41.221 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:39:41.282 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:39:41.396 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:39:41.506 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:39:41.564 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:39:41.676 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:39:41.782 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T06:39:41.839 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T06:39:41.960 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T06:39:42.071 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T06:39:42.128 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T06:39:42.244 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T06:39:42.349 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T06:39:42.409 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T06:39:42.519 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T06:39:42.627 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T06:39:42.688 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T06:39:42.879 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T06:39:48.743 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-59 zones={test-zone3, test-zone2} 2021-02-11T06:39:48.744 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:39:48.744 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:39:48.744 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-60 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:39:48.793 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:39:48.793 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:39:49.049 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:39:49.041+0000 7f7900f99080 1 bucket sync caught up with source: 2021-02-11T06:39:49.049 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [00000000006.60.5, 00000000004.32.1, 00000000006.42.5, 00000000004.44.1, 00000000006.75.5, 00000000004.35.1, 00000000004.45.1, 00000000006.63.5, 00000000004.64.1, 00000000004.59.1, 00000000004.46.1] 2021-02-11T06:39:49.049 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [00000000006.60.5, 00000000004.32.1, 00000000006.42.5, 00000000004.44.1, 00000000006.75.5, 00000000004.35.1, 00000000004.45.1, 00000000006.63.5, 00000000004.64.1, 00000000004.59.1, 00000000004.46.1] 2021-02-11T06:39:49.049 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:39:49.041+0000 7f7900f99080 0 bucket checkpoint complete 2021-02-11T06:39:49.081 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:39:49.081 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:39:49.081 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-60 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:39:49.129 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:39:49.129 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:39:49.368 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:39:49.365+0000 7f88c7a9f080 1 bucket sync caught up with source: 2021-02-11T06:39:49.368 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [00000000006.60.5, 00000000004.32.1, 00000000006.42.5, 00000000004.44.1, 00000000006.75.5, 00000000004.35.1, 00000000004.45.1, 00000000006.63.5, 00000000004.64.1, 00000000004.59.1, 00000000004.46.1] 2021-02-11T06:39:49.368 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [00000000006.60.5, 00000000004.32.1, 00000000006.42.5, 00000000004.44.1, 00000000006.75.5, 00000000004.35.1, 00000000004.45.1, 00000000006.63.5, 00000000004.64.1, 00000000004.59.1, 00000000004.46.1] 2021-02-11T06:39:49.368 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:39:49.365+0000 7f88c7a9f080 0 bucket checkpoint complete 2021-02-11T06:39:49.391 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:39:49.391 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:39:49.391 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-60 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:39:49.440 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:39:49.440 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:39:49.670 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:39:49.664+0000 7fdce9807080 1 bucket sync caught up with source: 2021-02-11T06:39:49.670 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [00000000006.60.5, 00000000004.32.1, 00000000006.42.5, 00000000004.44.1, 00000000006.75.5, 00000000004.35.1, 00000000004.45.1, 00000000006.63.5, 00000000004.64.1, 00000000004.59.1, 00000000004.46.1] 2021-02-11T06:39:49.670 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [00000000006.60.5, 00000000004.32.1, 00000000006.42.5, 00000000004.44.1, 00000000006.75.5, 00000000004.35.1, 00000000004.45.1, 00000000006.63.5, 00000000004.64.1, 00000000004.59.1, 00000000004.46.1] 2021-02-11T06:39:49.671 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:39:49.664+0000 7fdce9807080 0 bucket checkpoint complete 2021-02-11T06:39:49.699 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-60', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:39:49.699 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-60', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:39:49.700 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-60 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:39:49.747 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:39:49.748 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:39:50.011 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:39:50.009+0000 7f1e0d07d080 1 bucket sync caught up with source: 2021-02-11T06:39:50.011 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [00000000002.80.5, , 00000000002.42.5, , 00000000002.69.5, , , 00000000002.55.5, , , ] 2021-02-11T06:39:50.012 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [00000000002.80.5, , 00000000002.42.5, , 00000000002.69.5, , , 00000000002.55.5, , , ] 2021-02-11T06:39:50.012 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:39:50.009+0000 7f1e0d07d080 0 bucket checkpoint complete 2021-02-11T06:39:50.046 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-60', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:39:50.046 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-60', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:39:50.047 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-60 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:39:50.091 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:39:50.091 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:39:50.358 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:39:50.357+0000 7fa40661f080 1 bucket sync caught up with source: 2021-02-11T06:39:50.358 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [00000000002.80.5, , 00000000002.42.5, , 00000000002.69.5, , , 00000000002.55.5, , , ] 2021-02-11T06:39:50.358 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [00000000002.80.5, , 00000000002.42.5, , 00000000002.69.5, , , 00000000002.55.5, , , ] 2021-02-11T06:39:50.358 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:39:50.357+0000 7fa40661f080 0 bucket checkpoint complete 2021-02-11T06:39:50.392 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-60', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:39:50.392 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-60', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:39:50.392 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-60 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:39:50.479 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:39:50.479 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:39:50.744 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:39:50.741+0000 7f20a12b8080 1 bucket sync caught up with source: 2021-02-11T06:39:50.745 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [00000000002.53.5, , 00000000002.41.5, , 00000000002.70.5, , , 00000000002.50.5, , , ] 2021-02-11T06:39:50.745 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [00000000002.53.5, , 00000000002.41.5, , 00000000002.70.5, , , 00000000002.50.5, , , ] 2021-02-11T06:39:50.745 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:39:50.741+0000 7f20a12b8080 0 bucket checkpoint complete 2021-02-11T06:39:50.770 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-60', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:39:50.770 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-60', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:39:50.770 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-60 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:39:50.819 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:39:50.820 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:39:51.077 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:39:51.068+0000 7f65ab132080 1 bucket sync caught up with source: 2021-02-11T06:39:51.077 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [00000000002.53.5, , 00000000002.41.5, , 00000000002.70.5, , , 00000000002.50.5, , , ] 2021-02-11T06:39:51.077 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [00000000002.53.5, , 00000000002.41.5, , 00000000002.70.5, , , 00000000002.50.5, , , ] 2021-02-11T06:39:51.077 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:39:51.068+0000 7f65ab132080 0 bucket checkpoint complete 2021-02-11T06:39:51.110 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-60 zones={test-zone2, test-zone1} 2021-02-11T06:39:51.122 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:39:51.130 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T06:39:51.130 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T06:39:51.130 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T06:39:51.130 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T06:39:51.130 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T06:39:51.130 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T06:39:51.130 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T06:39:51.130 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T06:39:51.130 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:39:51.137 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T06:39:51.137 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T06:39:51.137 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T06:39:51.137 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T06:39:51.138 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T06:39:51.138 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T06:39:51.138 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T06:39:51.138 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T06:39:51.151 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:39:51.202 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:39:51.317 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:39:51.426 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:39:51.485 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:39:51.597 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:39:51.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:39:51.756 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:39:51.869 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:39:51.979 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:39:52.037 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:39:52.152 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:39:52.258 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T06:39:52.316 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T06:39:52.428 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T06:39:52.534 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T06:39:52.588 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T06:39:52.700 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T06:39:52.810 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T06:39:52.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T06:39:52.989 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T06:39:53.098 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T06:39:53.152 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T06:39:53.267 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T06:39:53.378 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-60 zones={test-zone2, test-zone1} 2021-02-11T06:39:53.378 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-60 zones={test-zone3, test-zone1} 2021-02-11T06:39:53.387 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:39:53.391 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T06:39:53.391 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T06:39:53.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T06:39:53.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T06:39:53.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T06:39:53.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T06:39:53.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T06:39:53.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T06:39:53.392 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:39:53.397 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T06:39:53.397 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T06:39:53.397 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T06:39:53.397 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T06:39:53.397 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T06:39:53.397 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T06:39:53.397 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T06:39:53.398 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T06:39:53.407 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:39:53.461 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:39:53.580 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:39:53.705 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:39:53.785 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:40:15.839 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:40:15.941 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:40:15.997 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:40:16.106 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:40:16.209 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:40:16.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:40:16.381 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:40:16.490 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T06:40:16.545 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T06:40:16.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T06:40:16.766 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T06:40:16.820 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T06:40:16.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T06:40:17.079 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T06:40:17.133 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T06:40:17.248 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T06:40:17.354 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T06:40:17.408 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T06:40:17.519 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T06:40:17.663 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-60 zones={test-zone3, test-zone1} 2021-02-11T06:40:17.663 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-60 zones={test-zone3, test-zone2} 2021-02-11T06:40:17.681 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:40:17.693 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T06:40:17.693 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T06:40:17.693 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T06:40:17.693 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T06:40:17.694 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T06:40:17.694 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T06:40:17.694 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T06:40:17.694 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T06:40:17.694 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:40:17.709 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T06:40:17.709 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T06:40:17.709 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T06:40:17.709 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T06:40:17.709 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T06:40:17.709 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T06:40:17.710 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T06:40:17.710 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T06:40:17.732 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:40:42.078 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:40:42.188 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:40:42.291 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:40:42.344 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:40:42.451 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:40:42.557 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:40:42.616 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:40:42.728 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:40:42.833 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:40:42.892 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:40:43.004 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:40:43.110 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T06:40:43.163 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T06:40:43.305 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T06:40:46.802 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T06:40:46.855 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T06:40:46.975 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T06:40:47.077 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T06:40:47.131 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T06:40:47.244 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T06:40:47.350 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T06:40:47.407 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T06:40:47.520 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T06:40:47.626 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-60 zones={test-zone3, test-zone2} 2021-02-11T06:40:47.626 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:40:47.626 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:40:47.627 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-61 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:40:47.670 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:40:47.671 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:40:47.945 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:40:47.940+0000 7ffad209d080 1 bucket sync caught up with source: 2021-02-11T06:40:47.946 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [00000000006.77.5, 00000000004.61.1, 00000000006.44.5, 00000000004.38.1, 00000000006.67.5, 00000000004.27.1, 00000000004.50.1, 00000000006.46.5, 00000000004.41.1, 00000000004.73.1, 00000000004.51.1] 2021-02-11T06:40:47.946 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [00000000006.77.5, 00000000004.61.1, 00000000006.44.5, 00000000004.38.1, 00000000006.67.5, 00000000004.27.1, 00000000004.50.1, 00000000006.46.5, 00000000004.41.1, 00000000004.73.1, 00000000004.51.1] 2021-02-11T06:40:47.946 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:40:47.940+0000 7ffad209d080 0 bucket checkpoint complete 2021-02-11T06:40:47.975 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:40:47.976 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:40:47.976 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-61 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:40:48.023 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:40:48.023 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:40:48.272 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:40:48.268+0000 7f4b50276080 1 bucket sync caught up with source: 2021-02-11T06:40:49.781 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [00000000006.77.5, 00000000004.61.1, 00000000006.44.5, 00000000004.38.1, 00000000006.67.5, 00000000004.27.1, 00000000004.50.1, 00000000006.46.5, 00000000004.41.1, 00000000004.73.1, 00000000004.51.1] 2021-02-11T06:40:49.782 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [00000000006.77.5, 00000000004.61.1, 00000000006.44.5, 00000000004.38.1, 00000000006.67.5, 00000000004.27.1, 00000000004.50.1, 00000000006.46.5, 00000000004.41.1, 00000000004.73.1, 00000000004.51.1] 2021-02-11T06:40:49.782 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:40:48.268+0000 7f4b50276080 0 bucket checkpoint complete 2021-02-11T06:40:49.783 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:40:49.783 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:40:49.783 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-61 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:40:49.831 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:40:49.832 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:40:50.067 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:40:50.060+0000 7f1db5a1e080 1 bucket sync caught up with source: 2021-02-11T06:40:50.067 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [00000000006.77.5, 00000000004.61.1, 00000000006.44.5, 00000000004.38.1, 00000000006.67.5, 00000000004.27.1, 00000000004.50.1, 00000000006.46.5, 00000000004.41.1, 00000000004.73.1, 00000000004.51.1] 2021-02-11T06:40:50.067 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [00000000006.77.5, 00000000004.61.1, 00000000006.44.5, 00000000004.38.1, 00000000006.67.5, 00000000004.27.1, 00000000004.50.1, 00000000006.46.5, 00000000004.41.1, 00000000004.73.1, 00000000004.51.1] 2021-02-11T06:40:50.068 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:40:50.060+0000 7f1db5a1e080 0 bucket checkpoint complete 2021-02-11T06:40:50.099 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-61', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:40:50.099 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-61', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:40:50.099 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-61 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:40:50.146 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:40:50.146 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:40:50.400 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:40:50.396+0000 7fbf5b4e3080 1 bucket sync caught up with source: 2021-02-11T06:40:50.401 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [00000000003.73.5, , 00000000002.36.5, , 00000000002.69.5, , , 00000000002.38.5, , , ] 2021-02-11T06:40:50.401 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [00000000003.73.5, , 00000000002.36.5, , 00000000002.69.5, , , 00000000002.38.5, , , ] 2021-02-11T06:40:50.401 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:40:50.396+0000 7fbf5b4e3080 0 bucket checkpoint complete 2021-02-11T06:40:50.435 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-61', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:40:50.435 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-61', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:40:50.435 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-61 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:40:50.479 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:40:50.479 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:40:50.748 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:40:50.744+0000 7fe5ba1ca080 1 bucket sync caught up with source: 2021-02-11T06:40:50.748 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [00000000003.73.5, , 00000000002.36.5, , 00000000002.69.5, , , 00000000002.38.5, , , ] 2021-02-11T06:40:50.748 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [00000000003.73.5, , 00000000002.36.5, , 00000000002.69.5, , , 00000000002.38.5, , , ] 2021-02-11T06:40:50.748 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:40:50.744+0000 7fe5ba1ca080 0 bucket checkpoint complete 2021-02-11T06:40:50.779 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-61', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:40:50.780 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-61', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:40:50.780 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-61 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:40:50.868 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:40:50.869 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:40:51.134 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:40:51.132+0000 7f909751a080 1 bucket sync caught up with source: 2021-02-11T06:40:51.135 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [00000000002.72.5, , 00000000002.34.5, , 00000000002.94.5, , , 00000000002.36.5, , , ] 2021-02-11T06:40:51.135 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [00000000002.72.5, , 00000000002.34.5, , 00000000002.94.5, , , 00000000002.36.5, , , ] 2021-02-11T06:40:51.135 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:40:51.132+0000 7f909751a080 0 bucket checkpoint complete 2021-02-11T06:40:51.158 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-61', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:40:51.158 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-61', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:40:51.159 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-61 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:40:51.208 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:40:51.208 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:40:51.449 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:40:51.443+0000 7f966d0a5080 1 bucket sync caught up with source: 2021-02-11T06:40:51.450 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [00000000002.72.5, , 00000000002.34.5, , 00000000002.94.5, , , 00000000002.36.5, , , ] 2021-02-11T06:40:51.450 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [00000000002.72.5, , 00000000002.34.5, , 00000000002.94.5, , , 00000000002.36.5, , , ] 2021-02-11T06:40:51.450 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:40:51.443+0000 7f966d0a5080 0 bucket checkpoint complete 2021-02-11T06:40:51.484 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-61 zones={test-zone2, test-zone1} 2021-02-11T06:40:51.494 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:40:51.500 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T06:40:51.500 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T06:40:51.501 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T06:40:51.501 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T06:40:51.501 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T06:40:51.501 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T06:40:51.501 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T06:40:51.501 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T06:40:51.501 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:40:51.508 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T06:40:51.508 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T06:40:51.508 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T06:40:51.508 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T06:40:51.508 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T06:40:51.508 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T06:40:51.508 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T06:40:51.509 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T06:40:51.520 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:40:51.575 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:40:51.693 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:40:51.798 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:40:51.853 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:40:51.964 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:40:52.070 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:40:52.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:40:52.233 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:40:52.338 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:40:52.392 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:40:52.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:40:52.614 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T06:40:52.669 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T06:40:52.785 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T06:40:52.898 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T06:40:52.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T06:40:53.063 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T06:40:53.175 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T06:41:16.081 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T06:41:16.191 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T06:41:16.297 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T06:41:16.353 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T06:41:16.464 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T06:41:16.571 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-61 zones={test-zone2, test-zone1} 2021-02-11T06:41:16.571 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-61 zones={test-zone3, test-zone1} 2021-02-11T06:41:16.580 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:41:16.585 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T06:41:16.585 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T06:41:16.585 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T06:41:16.585 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T06:41:16.585 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T06:41:16.585 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T06:41:16.585 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T06:41:16.585 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T06:41:16.585 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:41:16.590 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T06:41:16.590 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T06:41:16.590 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T06:41:16.590 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T06:41:16.590 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T06:41:16.591 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T06:41:16.591 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T06:41:16.591 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T06:41:16.601 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:41:16.656 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:41:16.776 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:41:16.885 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:41:16.939 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:41:17.047 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:41:17.155 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:41:17.212 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:41:17.324 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:41:17.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:41:17.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:41:17.595 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:41:17.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T06:41:17.756 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T06:41:17.877 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T06:41:17.981 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T06:41:18.036 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T06:41:18.153 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T06:41:18.258 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T06:41:18.312 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T06:41:18.427 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T06:41:18.579 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T06:41:18.637 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T06:41:18.747 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T06:41:18.913 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-61 zones={test-zone3, test-zone1} 2021-02-11T06:41:31.529 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-61 zones={test-zone3, test-zone2} 2021-02-11T06:41:31.540 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:41:31.547 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T06:41:31.547 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T06:41:31.547 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T06:41:31.547 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T06:41:31.547 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T06:41:31.548 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T06:41:31.548 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T06:41:31.548 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T06:41:31.548 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:41:31.554 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T06:41:31.554 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T06:41:31.554 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T06:41:31.554 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T06:41:31.554 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T06:41:31.555 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T06:41:31.555 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T06:41:31.555 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T06:41:31.567 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:41:31.620 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:41:31.741 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:41:31.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:41:31.916 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:41:32.044 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:41:32.370 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:41:32.423 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:41:32.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:41:32.642 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:41:32.696 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:41:32.808 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:41:32.918 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T06:41:32.973 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T06:41:33.089 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T06:41:33.194 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T06:41:33.248 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T06:41:33.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T06:41:33.474 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T06:41:33.541 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T06:41:33.657 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T06:41:33.773 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T06:41:33.832 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T06:41:33.948 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T06:41:34.054 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-61 zones={test-zone3, test-zone2} 2021-02-11T06:41:34.055 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:41:34.055 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:41:34.055 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:41:34.094 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:41:34.094 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:41:34.377 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"00000000000000000000:00000000000000012195","last_update":"2021-02-11T06:38:47.217381Z"},{"marker":"00000000000000000000:00000000000000008761","last_update":"2021-02-11T06:38:47.882979Z"},{"marker":"00000000000000000000:00000000000000008081","last_update":"2021-02-11T06:38:48.146434Z"},{"marker":"00000000000000000000:00000000000000012328","last_update":"2021-02-11T06:38:48.066184Z"}] 2021-02-11T06:41:34.377 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-02-11T06:41:34.377 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:41:34.377 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:41:34.378 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:41:34.420 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:41:34.420 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:41:34.688 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":15153533235123629405},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000012195","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:38:47.217381Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000008761","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:38:47.882979Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000008081","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:38:48.146434Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000012328","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:38:48.066184Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T06:41:34.688 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-02-11T06:41:34.688 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:41:34.688 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:41:34.688 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:41:34.729 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:41:34.729 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:41:34.999 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"00000000000000000000:00000000000000012195","last_update":"2021-02-11T06:38:47.217381Z"},{"marker":"00000000000000000000:00000000000000008761","last_update":"2021-02-11T06:38:47.882979Z"},{"marker":"00000000000000000000:00000000000000008081","last_update":"2021-02-11T06:38:48.146434Z"},{"marker":"00000000000000000000:00000000000000012328","last_update":"2021-02-11T06:38:48.066184Z"}] 2021-02-11T06:41:34.999 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-02-11T06:41:35.000 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:41:35.000 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:41:35.000 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:41:35.083 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:41:35.083 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:41:35.353 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":15040254564068855740},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000012195","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:38:47.217381Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000008761","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:38:47.882979Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000008081","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:38:48.146434Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000012328","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:38:48.066184Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T06:41:35.354 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-02-11T06:41:35.354 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:41:35.354 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:41:35.354 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:41:35.434 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:41:35.434 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:41:35.725 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"00000000000000000000:00000000000000012195","last_update":"2021-02-11T06:38:47.217381Z"},{"marker":"00000000000000000000:00000000000000008761","last_update":"2021-02-11T06:38:47.882979Z"},{"marker":"00000000000000000000:00000000000000008081","last_update":"2021-02-11T06:38:48.146434Z"},{"marker":"00000000000000000000:00000000000000012328","last_update":"2021-02-11T06:38:48.066184Z"}] 2021-02-11T06:41:35.725 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-02-11T06:41:35.725 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:41:35.726 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:41:35.726 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:41:35.768 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:41:35.769 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:41:36.034 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":1899655024930128290},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000012195","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:38:47.217381Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000008761","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:38:47.882979Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000008081","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:38:48.146434Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000012328","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:38:48.066184Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T06:41:36.035 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-02-11T06:41:36.040 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:41:36.040 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:41:36.040 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:41:36.116 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:41:36.116 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:41:36.363 INFO:teuthology.orchestra.run.smithi172.stdout:[{"marker":"00000000000000000000:00000000000000006220","last_update":"2021-02-11T06:36:45.572017Z"},{"marker":"00000000000000000000:00000000000000002770","last_update":"2021-02-11T06:38:47.862191Z"},{"marker":"00000000000000000000:00000000000000003964","last_update":"2021-02-11T06:38:48.101600Z"},{"marker":"00000000000000000000:00000000000000006485","last_update":"2021-02-11T06:38:51.260304Z"}] 2021-02-11T06:41:36.363 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-02-11T06:41:36.363 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:41:36.363 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:41:36.364 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:41:36.407 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:41:36.407 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:41:36.701 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":3699056310506273018},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000006220","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:36:45.572017Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002770","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:38:47.862191Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003964","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:38:48.101600Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000006485","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:38:51.260304Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T06:41:36.701 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-02-11T06:41:36.701 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:41:36.701 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:41:36.702 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:41:36.741 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:41:36.741 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:41:37.008 INFO:teuthology.orchestra.run.smithi172.stdout:[{"marker":"00000000000000000000:00000000000000006220","last_update":"2021-02-11T06:36:45.572017Z"},{"marker":"00000000000000000000:00000000000000002770","last_update":"2021-02-11T06:38:47.862191Z"},{"marker":"00000000000000000000:00000000000000003964","last_update":"2021-02-11T06:38:48.101600Z"},{"marker":"00000000000000000000:00000000000000006485","last_update":"2021-02-11T06:38:51.260304Z"}] 2021-02-11T06:41:37.009 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-02-11T06:41:37.009 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:41:37.009 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:41:37.009 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:41:37.053 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:41:37.053 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:41:37.335 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":995053126825359944},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000006220","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:36:45.572017Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002770","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:38:47.862191Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003964","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:38:48.101600Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000006485","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:38:51.260304Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T06:41:57.722 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-02-11T06:41:57.722 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:41:57.722 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:41:57.722 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:41:57.764 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:41:57.765 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:41:58.056 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"00000000000000000000:00000000000000006088","last_update":"2021-02-11T06:36:45.563961Z"},{"marker":"00000000000000000000:00000000000000002770","last_update":"2021-02-11T06:38:47.804246Z"},{"marker":"00000000000000000000:00000000000000003964","last_update":"2021-02-11T06:38:47.841922Z"},{"marker":"00000000000000000000:00000000000000006352","last_update":"2021-02-11T06:38:48.717382Z"}] 2021-02-11T06:41:58.057 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-02-11T06:41:58.057 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:41:58.057 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:41:58.057 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:41:58.144 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:41:58.144 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:41:58.431 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":398085786125260623},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000006088","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:36:45.563961Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002770","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:38:47.804246Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003964","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:38:47.841922Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000006352","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:38:48.717382Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T06:41:58.432 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-02-11T06:41:58.432 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:41:58.432 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:41:58.432 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:41:58.513 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:41:58.513 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:41:58.794 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"00000000000000000000:00000000000000006088","last_update":"2021-02-11T06:36:45.563961Z"},{"marker":"00000000000000000000:00000000000000002770","last_update":"2021-02-11T06:38:47.804246Z"},{"marker":"00000000000000000000:00000000000000003964","last_update":"2021-02-11T06:38:47.841922Z"},{"marker":"00000000000000000000:00000000000000006352","last_update":"2021-02-11T06:38:48.717382Z"}] 2021-02-11T06:42:08.687 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-02-11T06:42:08.688 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:42:08.688 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:42:08.688 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:42:08.734 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:42:08.734 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:42:09.018 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":11837782300501369896},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000006088","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:36:45.563961Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002770","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:38:47.804246Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003964","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:38:47.841922Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000006352","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:38:48.717382Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T06:42:09.018 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-02-11T06:42:09.019 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_sync_enable_right_after_disable ... ok 2021-02-11T06:42:09.019 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xscaju-62 2021-02-11T06:42:09.064 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xscaju-63 2021-02-11T06:42:09.144 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xscaju-64 2021-02-11T06:42:09.272 INFO:tasks.rgw.c1.client.1.smithi061.stdout:2021-02-11T06:42:09.266+0000 7f701fff7700 -1 data sync zone:5f91a5b1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T06:42:09.803 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:42:09.803 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:42:09.803 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:42:09.843 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:42:09.843 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:42:10.087 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=0d061ff0-ede8-4baa-beef-d975d317d505 2021-02-11T06:42:10.121 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613025729.185794_270.1","last_update":"2021-02-11T06:42:09.185794Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:42:10.121 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T06:42:10.121 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:42:10.121 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:42:10.122 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:42:10.155 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:42:10.156 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:42:10.417 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613025729.185794_270.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:42:09.185794Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:42:10.418 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T06:42:10.418 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T06:42:10.418 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:42:10.418 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:42:10.418 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:42:10.459 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:42:10.460 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:42:10.720 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613025729.185794_270.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:42:09.185794Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:42:10.721 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T06:42:10.721 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:42:10.721 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:42:10.721 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:42:10.721 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:42:10.763 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:42:10.764 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:42:11.007 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613025729.185794_270.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:42:09.185794Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:42:11.008 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:42:11.008 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:42:11.008 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:42:11.008 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-62 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:42:11.094 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:42:11.094 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:42:11.331 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:42:11.326+0000 7fda6bbe1080 1 bucket sync caught up with source: 2021-02-11T06:42:11.331 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [00000000001.29.5, , 00000000001.63.5, , 00000000001.50.5, , , 00000000001.31.5, , , ] 2021-02-11T06:42:11.332 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [00000000001.29.5, , 00000000001.63.5, , 00000000001.50.5, , , 00000000001.31.5, , , ] 2021-02-11T06:42:11.332 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:42:11.326+0000 7fda6bbe1080 0 bucket checkpoint complete 2021-02-11T06:42:11.361 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:42:11.361 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:42:11.361 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-62 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:42:11.411 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:42:11.411 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:42:11.673 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:42:11.670+0000 7fb4e8a1c080 1 bucket sync caught up with source: 2021-02-11T06:42:11.673 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [00000000001.29.5, , 00000000001.63.5, , 00000000001.50.5, , , 00000000001.31.5, , , ] 2021-02-11T06:42:11.673 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [00000000001.29.5, , 00000000001.63.5, , 00000000001.50.5, , , 00000000001.31.5, , , ] 2021-02-11T06:42:11.673 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:42:11.670+0000 7fb4e8a1c080 0 bucket checkpoint complete 2021-02-11T06:42:11.702 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:42:11.702 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:42:11.703 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-62 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:42:11.750 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:42:11.750 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:42:12.009 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:42:12.002+0000 7f6811b86080 1 bucket sync caught up with source: 2021-02-11T06:42:12.009 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [00000000001.29.5, , 00000000001.63.5, , 00000000001.50.5, , , 00000000001.31.5, , , ] 2021-02-11T06:42:12.009 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [00000000001.29.5, , 00000000001.63.5, , 00000000001.50.5, , , 00000000001.31.5, , , ] 2021-02-11T06:42:12.009 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:42:12.002+0000 7f6811b86080 0 bucket checkpoint complete 2021-02-11T06:42:12.039 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-62', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:42:12.039 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-62', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:42:12.040 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-62 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:42:12.089 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:42:12.089 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:42:12.347 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:42:12.342+0000 7f2c7e67e080 1 bucket sync caught up with source: 2021-02-11T06:42:12.347 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [00000000001.33.5, , 00000000001.62.5, , 00000000001.49.5, , , 00000000001.32.5, , , ] 2021-02-11T06:42:12.347 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [00000000001.33.5, , 00000000001.62.5, , 00000000001.49.5, , , 00000000001.32.5, , , ] 2021-02-11T06:42:12.348 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:42:12.342+0000 7f2c7e67e080 0 bucket checkpoint complete 2021-02-11T06:42:12.381 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-62', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:42:12.381 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-62', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:42:12.381 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-62 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:42:12.426 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:42:12.426 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:42:12.682 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:42:12.678+0000 7fda43102080 1 bucket sync caught up with source: 2021-02-11T06:42:12.682 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [00000000001.33.5, , 00000000001.62.5, , 00000000001.49.5, , , 00000000001.32.5, , , ] 2021-02-11T06:42:12.682 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [00000000001.33.5, , 00000000001.62.5, , 00000000001.49.5, , , 00000000001.32.5, , , ] 2021-02-11T06:42:12.682 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:42:12.678+0000 7fda43102080 0 bucket checkpoint complete 2021-02-11T06:42:12.720 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-62', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:42:12.721 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-62', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:42:12.721 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-62 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:42:12.768 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:42:12.768 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:42:13.039 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:42:13.034+0000 7fa6ea343080 1 bucket sync caught up with source: 2021-02-11T06:42:13.039 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [00000000001.40.5, , 00000000001.61.5, , 00000000001.50.5, , , 00000000001.30.5, , , ] 2021-02-11T06:42:13.039 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [00000000001.40.5, , 00000000001.61.5, , 00000000001.50.5, , , 00000000001.30.5, , , ] 2021-02-11T06:42:13.039 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:42:13.034+0000 7fa6ea343080 0 bucket checkpoint complete 2021-02-11T06:42:13.067 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-62', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:42:13.067 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-62', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:42:13.068 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-62 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:42:13.115 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:42:13.115 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:42:13.367 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:42:13.362+0000 7f8d83cdc080 1 bucket sync caught up with source: 2021-02-11T06:42:13.381 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [00000000001.40.5, , 00000000001.61.5, , 00000000001.50.5, , , 00000000001.30.5, , , ] 2021-02-11T06:42:13.381 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [00000000001.40.5, , 00000000001.61.5, , 00000000001.50.5, , , 00000000001.30.5, , , ] 2021-02-11T06:42:13.381 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:42:13.362+0000 7f8d83cdc080 0 bucket checkpoint complete 2021-02-11T06:42:13.403 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-62 zones={test-zone2, test-zone1} 2021-02-11T06:42:13.413 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:42:13.419 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T06:42:13.419 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T06:42:13.419 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T06:42:13.419 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T06:42:13.420 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:42:13.426 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T06:42:13.426 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T06:42:13.426 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T06:42:13.426 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T06:42:13.441 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:42:13.506 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:42:13.660 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:42:13.790 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:42:13.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:42:13.972 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:42:14.078 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:42:14.136 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:42:14.247 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:42:14.353 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:42:14.412 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:42:14.522 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:42:14.626 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-62 zones={test-zone2, test-zone1} 2021-02-11T06:42:14.626 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-62 zones={test-zone3, test-zone1} 2021-02-11T06:42:14.635 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:42:14.640 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T06:42:14.640 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T06:42:14.640 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T06:42:14.640 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T06:42:14.640 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:42:14.644 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T06:42:14.644 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T06:42:14.644 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T06:42:14.644 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T06:42:14.654 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:42:14.708 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:42:14.915 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:42:15.019 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:42:15.077 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:42:15.189 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:42:15.297 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:42:15.355 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:42:15.471 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:42:15.578 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:42:15.632 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:42:15.748 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:42:15.850 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-62 zones={test-zone3, test-zone1} 2021-02-11T06:42:15.851 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-62 zones={test-zone3, test-zone2} 2021-02-11T06:42:15.862 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:42:15.911 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T06:42:15.911 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T06:42:15.911 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T06:42:15.911 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T06:42:15.911 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:42:15.918 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T06:42:15.918 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T06:42:15.918 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T06:42:15.918 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T06:42:15.974 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:42:16.032 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:42:16.144 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:42:16.258 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:42:16.311 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:42:16.424 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:42:16.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:42:16.593 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:42:16.708 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:42:16.818 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:42:16.872 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:42:16.983 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:42:17.095 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-62 zones={test-zone3, test-zone2} 2021-02-11T06:42:17.095 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:42:17.095 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:42:17.095 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-63 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:42:17.138 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:42:17.138 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:42:17.360 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:42:17.354+0000 7f5af4bee080 1 bucket sync caught up with source: 2021-02-11T06:42:17.360 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [00000000001.41.5, , 00000000001.35.5, , 00000000001.63.5, , , 00000000001.112.5, , , ] 2021-02-11T06:42:17.360 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [00000000001.41.5, , 00000000001.35.5, , 00000000001.63.5, , , 00000000001.112.5, , , ] 2021-02-11T06:42:17.360 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:42:17.354+0000 7f5af4bee080 0 bucket checkpoint complete 2021-02-11T06:42:17.390 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:42:17.390 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:42:17.390 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-63 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:42:17.431 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:42:17.431 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:42:17.697 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:42:17.694+0000 7fc327fce080 1 bucket sync caught up with source: 2021-02-11T06:42:17.698 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [00000000001.41.5, , 00000000001.35.5, , 00000000001.63.5, , , 00000000001.112.5, , , ] 2021-02-11T06:42:17.698 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [00000000001.41.5, , 00000000001.35.5, , 00000000001.63.5, , , 00000000001.112.5, , , ] 2021-02-11T06:42:17.698 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:42:17.694+0000 7fc327fce080 0 bucket checkpoint complete 2021-02-11T06:42:17.726 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:42:17.726 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:42:17.726 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-63 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:42:17.773 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:42:17.773 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:42:18.014 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:42:18.010+0000 7ff642665080 1 bucket sync caught up with source: 2021-02-11T06:42:18.014 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [00000000001.41.5, , 00000000001.35.5, , 00000000001.63.5, , , 00000000001.112.5, , , ] 2021-02-11T06:42:18.015 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [00000000001.41.5, , 00000000001.35.5, , 00000000001.63.5, , , 00000000001.112.5, , , ] 2021-02-11T06:42:18.015 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:42:18.010+0000 7ff642665080 0 bucket checkpoint complete 2021-02-11T06:42:18.046 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-63', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:42:18.046 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-63', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:42:18.046 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-63 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:42:18.094 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:42:18.094 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:42:18.362 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:42:18.358+0000 7f1b57b89080 1 bucket sync caught up with source: 2021-02-11T06:42:18.363 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [00000000001.40.5, , 00000000001.38.5, , 00000000001.86.5, , , 00000000001.73.5, , , ] 2021-02-11T06:42:18.363 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [00000000001.40.5, , 00000000001.38.5, , 00000000001.86.5, , , 00000000001.73.5, , , ] 2021-02-11T06:42:18.363 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:42:18.358+0000 7f1b57b89080 0 bucket checkpoint complete 2021-02-11T06:42:18.397 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-63', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:42:18.397 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-63', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:42:18.398 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-63 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:42:18.446 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:42:18.446 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:42:18.712 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:42:18.706+0000 7faf9abff080 1 bucket sync caught up with source: 2021-02-11T06:42:18.713 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [00000000001.40.5, , 00000000001.38.5, , 00000000001.86.5, , , 00000000001.73.5, , , ] 2021-02-11T06:42:18.713 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [00000000001.40.5, , 00000000001.38.5, , 00000000001.86.5, , , 00000000001.73.5, , , ] 2021-02-11T06:42:18.713 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:42:18.706+0000 7faf9abff080 0 bucket checkpoint complete 2021-02-11T06:42:18.744 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-63', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:42:18.744 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-63', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:42:18.744 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-63 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:42:18.793 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:42:18.794 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:42:19.067 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:42:19.062+0000 7f767d0c1080 1 waiting for incremental sync to catch up: 2021-02-11T06:42:19.067 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, , 00000000001.29.5, , 00000000001.58.5, , , , , , ] 2021-02-11T06:42:19.067 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [00000000001.43.5, , 00000000001.29.5, , 00000000001.58.5, , , 00000000001.75.5, , , ] 2021-02-11T06:42:24.071 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:42:24.066+0000 7f767d0c1080 1 bucket sync caught up with source: 2021-02-11T06:42:35.130 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [00000000001.43.5, , 00000000001.29.5, , 00000000001.58.5, , , 00000000001.75.5, , , ] 2021-02-11T06:42:35.130 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [00000000001.43.5, , 00000000001.29.5, , 00000000001.58.5, , , 00000000001.75.5, , , ] 2021-02-11T06:42:35.130 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:42:24.066+0000 7f767d0c1080 0 bucket checkpoint complete 2021-02-11T06:42:35.132 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-63', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:42:35.132 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-63', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:42:35.132 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-63 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:42:35.181 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:42:35.181 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:42:35.420 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:42:35.413+0000 7fe0a29cd080 1 bucket sync caught up with source: 2021-02-11T06:42:35.420 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [00000000001.43.5, , 00000000001.29.5, , 00000000001.58.5, , , 00000000001.75.5, , , ] 2021-02-11T06:42:35.420 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [00000000001.43.5, , 00000000001.29.5, , 00000000001.58.5, , , 00000000001.75.5, , , ] 2021-02-11T06:42:35.420 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:42:35.413+0000 7fe0a29cd080 0 bucket checkpoint complete 2021-02-11T06:42:35.453 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-63 zones={test-zone2, test-zone1} 2021-02-11T06:42:35.463 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:42:35.511 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T06:42:35.511 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T06:42:35.511 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T06:42:35.511 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T06:42:35.511 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:42:35.519 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T06:42:35.519 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T06:42:35.519 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T06:42:35.520 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T06:42:35.579 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:42:35.636 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:42:35.749 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:42:35.854 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:42:35.911 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:42:36.024 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:42:36.134 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:42:36.189 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:42:36.300 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:42:36.406 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:42:36.460 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:42:36.577 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:42:36.683 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-63 zones={test-zone2, test-zone1} 2021-02-11T06:42:36.684 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-63 zones={test-zone3, test-zone1} 2021-02-11T06:42:36.693 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:42:36.698 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T06:42:36.699 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T06:42:36.699 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T06:42:36.699 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T06:42:36.699 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:42:36.704 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T06:42:36.704 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T06:42:36.704 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T06:42:36.704 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T06:42:36.715 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:42:36.767 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:42:36.874 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:42:36.977 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:42:37.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:42:37.145 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:42:37.248 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:42:37.304 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:42:37.417 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:42:37.523 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:42:37.580 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:42:37.693 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:42:37.797 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-63 zones={test-zone3, test-zone1} 2021-02-11T06:42:37.798 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-63 zones={test-zone3, test-zone2} 2021-02-11T06:42:37.810 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:42:37.815 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T06:42:37.815 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T06:42:37.815 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T06:42:37.815 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T06:42:37.815 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:42:37.821 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T06:42:37.821 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T06:42:37.821 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T06:42:37.821 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T06:42:37.831 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:42:37.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:42:37.996 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:42:38.102 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:42:38.156 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:42:38.269 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:42:38.384 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:42:51.966 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:42:52.074 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:42:52.177 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:42:52.232 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:42:52.346 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:42:52.450 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-63 zones={test-zone3, test-zone2} 2021-02-11T06:42:52.450 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:42:52.450 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:42:52.451 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-64 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:42:52.496 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:42:52.496 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:42:52.740 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:42:52.733+0000 7f48fd647080 1 bucket sync caught up with source: 2021-02-11T06:42:52.740 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [00000000001.53.5, , 00000000001.51.5, , 00000000001.33.5, , , 00000000001.39.5, , , ] 2021-02-11T06:42:52.740 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [00000000001.53.5, , 00000000001.51.5, , 00000000001.33.5, , , 00000000001.39.5, , , ] 2021-02-11T06:42:52.740 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:42:52.733+0000 7f48fd647080 0 bucket checkpoint complete 2021-02-11T06:42:52.774 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:42:52.774 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:42:52.774 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-64 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:42:52.822 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:42:52.822 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:42:53.082 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:42:53.081+0000 7fc59db7b080 1 bucket sync caught up with source: 2021-02-11T06:42:53.082 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [00000000001.53.5, , 00000000001.51.5, , 00000000001.33.5, , , 00000000001.39.5, , , ] 2021-02-11T06:42:53.082 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [00000000001.53.5, , 00000000001.51.5, , 00000000001.33.5, , , 00000000001.39.5, , , ] 2021-02-11T06:42:53.082 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:42:53.081+0000 7fc59db7b080 0 bucket checkpoint complete 2021-02-11T06:42:53.109 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:42:53.110 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:42:53.110 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-64 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:42:53.159 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:42:53.160 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:42:53.412 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:42:53.405+0000 7fdf32485080 1 bucket sync caught up with source: 2021-02-11T06:42:53.412 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [00000000001.53.5, , 00000000001.51.5, , 00000000001.33.5, , , 00000000001.39.5, , , ] 2021-02-11T06:42:53.412 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [00000000001.53.5, , 00000000001.51.5, , 00000000001.33.5, , , 00000000001.39.5, , , ] 2021-02-11T06:42:53.412 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:42:53.405+0000 7fdf32485080 0 bucket checkpoint complete 2021-02-11T06:42:53.444 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-64', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:42:53.444 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-64', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:42:53.445 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-64 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:42:53.493 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:42:53.493 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:42:53.765 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:42:53.761+0000 7fd95c412080 1 bucket sync caught up with source: 2021-02-11T06:42:53.876 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [00000000001.64.5, , 00000000001.55.5, , 00000000001.34.5, , , 00000000001.44.5, , , ] 2021-02-11T06:42:53.878 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [00000000001.64.5, , 00000000001.55.5, , 00000000001.34.5, , , 00000000001.44.5, , , ] 2021-02-11T06:42:53.878 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:42:53.761+0000 7fd95c412080 0 bucket checkpoint complete 2021-02-11T06:42:53.879 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-64', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:42:53.880 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-64', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:42:53.880 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-64 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:42:53.931 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:42:53.932 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:42:54.203 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:42:54.201+0000 7f9f2a38f080 1 bucket sync caught up with source: 2021-02-11T06:42:54.204 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [00000000001.64.5, , 00000000001.55.5, , 00000000001.34.5, , , 00000000001.44.5, , , ] 2021-02-11T06:42:54.204 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [00000000001.64.5, , 00000000001.55.5, , 00000000001.34.5, , , 00000000001.44.5, , , ] 2021-02-11T06:42:54.204 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:42:54.201+0000 7f9f2a38f080 0 bucket checkpoint complete 2021-02-11T06:42:54.235 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-64', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:42:54.235 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-64', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:42:54.235 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-64 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:42:54.282 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:42:54.282 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:42:54.541 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:42:54.537+0000 7fa932693080 1 bucket sync caught up with source: 2021-02-11T06:42:54.541 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [00000000001.59.5, , 00000000001.56.5, , 00000000001.31.5, , , 00000000001.44.5, , , ] 2021-02-11T06:42:54.541 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [00000000001.59.5, , 00000000001.56.5, , 00000000001.31.5, , , 00000000001.44.5, , , ] 2021-02-11T06:42:54.541 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:42:54.537+0000 7fa932693080 0 bucket checkpoint complete 2021-02-11T06:42:54.570 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-64', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:42:54.571 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-64', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:42:54.571 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-64 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:42:54.618 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:42:54.618 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:42:54.872 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:42:54.865+0000 7f5f9b639080 1 bucket sync caught up with source: 2021-02-11T06:42:54.873 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [00000000001.59.5, , 00000000001.56.5, , 00000000001.31.5, , , 00000000001.44.5, , , ] 2021-02-11T06:42:54.873 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [00000000001.59.5, , 00000000001.56.5, , 00000000001.31.5, , , 00000000001.44.5, , , ] 2021-02-11T06:42:54.873 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:42:54.865+0000 7f5f9b639080 0 bucket checkpoint complete 2021-02-11T06:42:54.905 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-64 zones={test-zone2, test-zone1} 2021-02-11T06:42:54.916 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:42:54.922 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T06:42:54.922 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T06:42:54.922 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T06:42:54.922 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T06:42:54.922 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:42:54.971 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T06:42:54.971 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T06:42:54.971 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T06:42:54.971 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T06:42:54.986 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:42:55.040 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:42:55.152 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:42:55.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:42:55.312 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:42:55.425 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:42:55.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:42:55.588 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:42:55.705 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:42:55.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:42:55.868 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:42:55.981 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:42:56.094 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-64 zones={test-zone2, test-zone1} 2021-02-11T06:42:56.095 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-64 zones={test-zone3, test-zone1} 2021-02-11T06:42:56.103 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:42:56.107 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T06:42:56.108 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T06:42:56.108 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T06:42:56.108 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T06:42:56.108 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:42:56.155 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T06:42:56.155 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T06:42:56.155 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T06:42:56.155 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T06:42:56.167 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:42:56.265 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:42:56.381 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:42:56.486 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:42:56.540 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:42:56.657 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:42:56.762 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:42:56.821 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:42:56.932 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:42:57.042 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:42:57.097 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:42:57.206 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:42:57.309 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-64 zones={test-zone3, test-zone1} 2021-02-11T06:42:57.309 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-64 zones={test-zone3, test-zone2} 2021-02-11T06:42:57.320 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:42:57.326 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T06:42:57.326 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T06:42:57.326 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T06:42:57.326 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T06:42:57.326 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:42:57.332 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T06:42:57.332 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T06:42:57.332 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T06:42:57.332 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T06:42:57.344 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:42:57.400 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:42:57.512 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:42:57.623 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:42:57.680 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:42:57.796 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:42:57.929 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:43:11.386 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:43:11.498 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:43:11.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:43:11.660 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:43:11.776 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:43:11.883 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-64 zones={test-zone3, test-zone2} 2021-02-11T06:43:11.884 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'xscaju-62', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:43:11.884 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'xscaju-62', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:43:11.884 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket xscaju-62 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T06:43:11.927 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:43:11.927 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:43:12.281 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'xscaju-63', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:43:12.281 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'xscaju-63', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:43:12.282 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket xscaju-63 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T06:43:12.366 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:43:12.366 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:43:12.710 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'xscaju-64', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:43:12.710 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'xscaju-64', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:43:12.710 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket xscaju-64 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T06:43:12.796 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:43:12.797 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:43:13.152 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:43:13.153 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:43:13.153 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:43:13.234 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:43:13.235 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:43:13.486 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=0d061ff0-ede8-4baa-beef-d975d317d505 2021-02-11T06:43:13.518 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613025793.102581_276.1","last_update":"2021-02-11T06:43:13.102581Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:43:13.518 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T06:43:13.518 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:43:13.519 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:43:13.519 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:43:13.559 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:43:13.559 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:43:13.827 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613025729.185794_270.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:42:09.185794Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:43:13.827 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613025793.102581_276.1 target=1_1613025729.185794_270.1 2021-02-11T06:43:18.828 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:43:25.809 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:43:25.809 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:43:25.851 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:43:25.851 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:43:26.136 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613025793.102581_276.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:43:13.102581Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:43:26.136 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T06:43:26.136 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T06:43:26.136 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:43:26.137 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:43:26.137 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:43:26.184 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:43:26.184 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:43:26.462 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613025793.102581_276.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:43:13.102581Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:43:26.463 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T06:43:26.463 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:43:26.463 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:43:26.463 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:43:26.463 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:43:26.503 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:43:26.503 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:43:26.767 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613025793.102581_276.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:43:13.102581Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:43:26.767 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:43:27.371 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'xscaju-62', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:43:27.371 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'xscaju-62', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:43:27.371 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket xscaju-62 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T06:43:27.411 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:43:27.411 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:43:27.723 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'xscaju-63', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:43:27.723 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'xscaju-63', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:43:27.723 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket xscaju-63 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T06:43:27.809 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:43:27.810 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:43:28.142 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'xscaju-64', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:43:36.811 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'xscaju-64', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:43:36.811 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket xscaju-64 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T06:43:36.852 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:43:36.853 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:43:37.205 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:43:37.205 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:43:37.205 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-62 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:43:37.256 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:43:37.257 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:43:37.490 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:43:37.484+0000 7f38990d4080 1 waiting for incremental sync to catch up: 2021-02-11T06:43:37.490 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [00000000006.42.1, , 00000000006.72.1, 00000000004.70.1, 00000000006.58.1, , 00000000004.51.1, 00000000006.41.1, 00000000004.82.1, , 00000000004.73.1] 2021-02-11T06:43:37.490 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [00000000006.42.1, 00000000004.56.1, 00000000006.72.1, 00000000004.70.1, 00000000006.58.1, 00000000004.43.1, 00000000004.51.1, 00000000006.41.1, 00000000004.82.1, 00000000004.65.1, 00000000004.73.1] 2021-02-11T06:43:42.495 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:43:42.488+0000 7f38990d4080 1 waiting for incremental sync to catch up: 2021-02-11T06:43:57.206 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [00000000006.42.1, , 00000000006.72.1, 00000000004.70.1, 00000000006.58.1, , 00000000004.51.1, 00000000006.41.1, 00000000004.82.1, , 00000000004.73.1] 2021-02-11T06:43:57.206 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [00000000006.42.1, 00000000004.56.1, 00000000006.72.1, 00000000004.70.1, 00000000006.58.1, 00000000004.43.1, 00000000004.51.1, 00000000006.41.1, 00000000004.82.1, 00000000004.65.1, 00000000004.73.1] 2021-02-11T06:43:57.207 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:43:47.492+0000 7f38990d4080 1 bucket sync caught up with source: 2021-02-11T06:43:57.207 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [00000000006.42.1, 00000000004.56.1, 00000000006.72.1, 00000000004.70.1, 00000000006.58.1, 00000000004.43.1, 00000000004.51.1, 00000000006.41.1, 00000000004.82.1, 00000000004.65.1, 00000000004.73.1] 2021-02-11T06:43:57.208 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [00000000006.42.1, 00000000004.56.1, 00000000006.72.1, 00000000004.70.1, 00000000006.58.1, 00000000004.43.1, 00000000004.51.1, 00000000006.41.1, 00000000004.82.1, 00000000004.65.1, 00000000004.73.1] 2021-02-11T06:43:57.208 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:43:47.492+0000 7f38990d4080 0 bucket checkpoint complete 2021-02-11T06:43:57.208 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:43:57.208 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:43:57.208 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-62 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:43:57.255 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:43:57.255 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:43:57.509 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:43:57.508+0000 7f99569e6080 1 bucket sync caught up with source: 2021-02-11T06:43:57.509 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [00000000006.42.1, 00000000004.56.1, 00000000006.72.1, 00000000004.70.1, 00000000006.58.1, 00000000004.43.1, 00000000004.51.1, 00000000006.41.1, 00000000004.82.1, 00000000004.65.1, 00000000004.73.1] 2021-02-11T06:43:57.510 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [00000000006.42.1, 00000000004.56.1, 00000000006.72.1, 00000000004.70.1, 00000000006.58.1, 00000000004.43.1, 00000000004.51.1, 00000000006.41.1, 00000000004.82.1, 00000000004.65.1, 00000000004.73.1] 2021-02-11T06:43:57.510 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:43:57.508+0000 7f99569e6080 0 bucket checkpoint complete 2021-02-11T06:43:57.535 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:43:57.536 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:43:57.536 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-62 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:43:57.585 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:43:57.585 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:43:57.830 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:43:57.824+0000 7f5ae428a080 1 bucket sync caught up with source: 2021-02-11T06:43:57.830 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [00000000006.42.1, 00000000004.56.1, 00000000006.72.1, 00000000004.70.1, 00000000006.58.1, 00000000004.43.1, 00000000004.51.1, 00000000006.41.1, 00000000004.82.1, 00000000004.65.1, 00000000004.73.1] 2021-02-11T06:43:57.831 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [00000000006.42.1, 00000000004.56.1, 00000000006.72.1, 00000000004.70.1, 00000000006.58.1, 00000000004.43.1, 00000000004.51.1, 00000000006.41.1, 00000000004.82.1, 00000000004.65.1, 00000000004.73.1] 2021-02-11T06:43:57.831 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:43:57.824+0000 7f5ae428a080 0 bucket checkpoint complete 2021-02-11T06:43:57.862 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-62', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:43:57.862 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-62', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:43:57.863 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-62 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:43:57.912 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:43:57.912 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:43:58.186 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:43:58.184+0000 7fe17921d080 1 bucket sync caught up with source: 2021-02-11T06:43:58.187 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [00000000004.47.5, 00000000002.53.1, 00000000004.72.5, 00000000002.84.1, 00000000004.57.5, 00000000002.43.1, 00000000002.50.1, 00000000004.42.5, 00000000002.82.1, 00000000002.66.1, 00000000002.68.1] 2021-02-11T06:43:58.187 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [00000000004.47.5, 00000000002.53.1, 00000000004.72.5, 00000000002.84.1, 00000000004.57.5, 00000000002.43.1, 00000000002.50.1, 00000000004.42.5, 00000000002.82.1, 00000000002.66.1, 00000000002.68.1] 2021-02-11T06:43:58.187 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:43:58.184+0000 7fe17921d080 0 bucket checkpoint complete 2021-02-11T06:43:58.211 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-62', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:43:58.211 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-62', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:43:58.211 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-62 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:43:58.256 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:43:58.256 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:43:58.516 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:43:58.512+0000 7fba1480b080 1 bucket sync caught up with source: 2021-02-11T06:43:58.516 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [00000000004.47.5, 00000000002.53.1, 00000000004.72.5, 00000000002.84.1, 00000000004.57.5, 00000000002.43.1, 00000000002.50.1, 00000000004.42.5, 00000000002.82.1, 00000000002.66.1, 00000000002.68.1] 2021-02-11T06:43:58.516 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [00000000004.47.5, 00000000002.53.1, 00000000004.72.5, 00000000002.84.1, 00000000004.57.5, 00000000002.43.1, 00000000002.50.1, 00000000004.42.5, 00000000002.82.1, 00000000002.66.1, 00000000002.68.1] 2021-02-11T06:43:58.516 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:43:58.512+0000 7fba1480b080 0 bucket checkpoint complete 2021-02-11T06:43:58.552 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-62', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:43:58.552 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-62', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:43:58.553 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-62 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:43:58.642 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:43:58.964 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:43:58.965 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:43:58.896+0000 7f263a6e0080 1 bucket sync caught up with source: 2021-02-11T06:43:58.965 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [00000000004.51.1, 00000000002.55.1, 00000000004.68.1, 00000000002.65.1, 00000000004.57.1, 00000000002.52.1, 00000000002.44.1, 00000000004.40.1, 00000000002.82.1, 00000000002.65.1, 00000000002.69.1] 2021-02-11T06:43:58.965 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [00000000004.51.1, 00000000002.55.1, 00000000004.68.1, 00000000002.65.1, 00000000004.57.1, 00000000002.52.1, 00000000002.44.1, 00000000004.40.1, 00000000002.82.1, 00000000002.65.1, 00000000002.69.1] 2021-02-11T06:43:58.966 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:43:58.896+0000 7f263a6e0080 0 bucket checkpoint complete 2021-02-11T06:43:58.966 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-62', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:43:58.966 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-62', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:43:58.966 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-62 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:43:59.017 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:43:59.017 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:43:59.275 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:43:59.268+0000 7f98341fb080 1 bucket sync caught up with source: 2021-02-11T06:43:59.275 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [00000000004.51.1, 00000000002.55.1, 00000000004.68.1, 00000000002.65.1, 00000000004.57.1, 00000000002.52.1, 00000000002.44.1, 00000000004.40.1, 00000000002.82.1, 00000000002.65.1, 00000000002.69.1] 2021-02-11T06:43:59.275 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [00000000004.51.1, 00000000002.55.1, 00000000004.68.1, 00000000002.65.1, 00000000004.57.1, 00000000002.52.1, 00000000002.44.1, 00000000004.40.1, 00000000002.82.1, 00000000002.65.1, 00000000002.69.1] 2021-02-11T06:43:59.275 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:43:59.268+0000 7f98341fb080 0 bucket checkpoint complete 2021-02-11T06:43:59.302 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-62 zones={test-zone2, test-zone1} 2021-02-11T06:43:59.313 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:43:59.319 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T06:43:59.319 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T06:43:59.319 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T06:43:59.319 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T06:43:59.319 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T06:43:59.319 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T06:43:59.319 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T06:43:59.319 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T06:43:59.319 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:43:59.328 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T06:43:59.328 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T06:43:59.329 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T06:43:59.329 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T06:43:59.329 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T06:43:59.329 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T06:43:59.329 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T06:43:59.329 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T06:43:59.342 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:43:59.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:43:59.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:43:59.605 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:43:59.659 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:43:59.778 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:43:59.882 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:43:59.936 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:44:00.051 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:44:00.154 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:44:00.207 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:44:00.315 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:44:00.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T06:44:00.476 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T06:44:00.588 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T06:44:00.694 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T06:44:00.748 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T06:44:00.860 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T06:44:00.970 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T06:44:01.023 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T06:44:01.137 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T06:44:01.243 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T06:44:01.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T06:44:01.408 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T06:44:01.519 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-62 zones={test-zone2, test-zone1} 2021-02-11T06:44:01.519 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-62 zones={test-zone3, test-zone1} 2021-02-11T06:44:01.528 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:44:01.532 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T06:44:01.533 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T06:44:01.533 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T06:44:01.533 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T06:44:01.533 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T06:44:01.533 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T06:44:01.533 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T06:44:01.533 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T06:44:01.533 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:44:01.538 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T06:44:01.538 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T06:44:01.538 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T06:44:01.538 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T06:44:01.538 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T06:44:01.538 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T06:44:01.538 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T06:44:01.538 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T06:44:01.548 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:44:01.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:44:01.716 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:44:01.821 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:44:01.875 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:44:01.988 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:44:02.093 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:44:02.148 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:44:02.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:44:02.365 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:44:02.419 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:44:02.530 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:44:02.634 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T06:44:02.688 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T06:44:02.799 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T06:44:02.901 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T06:44:02.956 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T06:44:03.059 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T06:44:03.161 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T06:44:03.218 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T06:44:03.326 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T06:44:16.589 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T06:44:16.643 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T06:44:16.755 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T06:44:16.857 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-62 zones={test-zone3, test-zone1} 2021-02-11T06:44:16.858 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-62 zones={test-zone3, test-zone2} 2021-02-11T06:44:16.870 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:44:16.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T06:44:16.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T06:44:16.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T06:44:16.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T06:44:16.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T06:44:16.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T06:44:16.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T06:44:16.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T06:44:16.875 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:44:16.879 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T06:44:16.880 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T06:44:16.880 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T06:44:16.880 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T06:44:16.880 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T06:44:16.880 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T06:44:16.880 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T06:44:16.880 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T06:44:16.891 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:44:16.943 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:44:17.056 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:44:17.161 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:44:17.216 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:44:17.332 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:44:17.438 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:44:17.492 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:44:17.604 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:44:17.720 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:44:26.977 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:44:27.088 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:44:27.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T06:44:27.245 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T06:44:27.356 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T06:44:27.459 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T06:44:27.511 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T06:44:27.624 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T06:44:27.730 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T06:44:27.783 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T06:44:27.895 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T06:44:28.002 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T06:44:28.056 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T06:44:28.167 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T06:44:28.270 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-62 zones={test-zone3, test-zone2} 2021-02-11T06:44:28.270 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:44:28.271 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:44:28.271 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-63 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:44:28.317 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:44:28.317 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:44:28.572 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:44:28.567+0000 7fd8012e5080 1 bucket sync caught up with source: 2021-02-11T06:44:28.572 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [00000000006.47.5, 00000000004.45.1, 00000000006.41.5, 00000000004.62.1, 00000000006.69.5, 00000000004.31.1, 00000000004.63.1, 00000000006.118.5, 00000000004.48.1, 00000000004.72.1, 00000000004.49.1] 2021-02-11T06:44:28.572 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [00000000006.47.5, 00000000004.45.1, 00000000006.41.5, 00000000004.62.1, 00000000006.69.5, 00000000004.31.1, 00000000004.63.1, 00000000006.118.5, 00000000004.48.1, 00000000004.72.1, 00000000004.49.1] 2021-02-11T06:44:28.572 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:44:28.567+0000 7fd8012e5080 0 bucket checkpoint complete 2021-02-11T06:44:28.603 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:44:28.603 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:44:28.604 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-63 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:44:28.650 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:44:28.650 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:44:28.937 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:44:28.936+0000 7fb9c66d2080 1 bucket sync caught up with source: 2021-02-11T06:44:29.611 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [00000000006.47.5, 00000000004.45.1, 00000000006.41.5, 00000000004.62.1, 00000000006.69.5, 00000000004.31.1, 00000000004.63.1, 00000000006.118.5, 00000000004.48.1, 00000000004.72.1, 00000000004.49.1] 2021-02-11T06:44:29.612 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [00000000006.47.5, 00000000004.45.1, 00000000006.41.5, 00000000004.62.1, 00000000006.69.5, 00000000004.31.1, 00000000004.63.1, 00000000006.118.5, 00000000004.48.1, 00000000004.72.1, 00000000004.49.1] 2021-02-11T06:44:29.612 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:44:28.936+0000 7fb9c66d2080 0 bucket checkpoint complete 2021-02-11T06:44:29.613 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:44:29.613 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:44:29.613 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-63 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:44:29.662 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:44:29.663 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:44:29.899 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:44:29.895+0000 7f2e778e4080 1 bucket sync caught up with source: 2021-02-11T06:44:29.899 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [00000000006.47.5, 00000000004.45.1, 00000000006.41.5, 00000000004.62.1, 00000000006.69.5, 00000000004.31.1, 00000000004.63.1, 00000000006.118.5, 00000000004.48.1, 00000000004.72.1, 00000000004.49.1] 2021-02-11T06:44:29.900 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [00000000006.47.5, 00000000004.45.1, 00000000006.41.5, 00000000004.62.1, 00000000006.69.5, 00000000004.31.1, 00000000004.63.1, 00000000006.118.5, 00000000004.48.1, 00000000004.72.1, 00000000004.49.1] 2021-02-11T06:44:29.900 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:44:29.895+0000 7f2e778e4080 0 bucket checkpoint complete 2021-02-11T06:44:29.933 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-63', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:44:29.933 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-63', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:44:29.933 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-63 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:44:29.980 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:44:29.980 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:44:30.242 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:44:30.240+0000 7f02a4440080 1 bucket sync caught up with source: 2021-02-11T06:44:30.243 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [00000000004.45.1, 00000000002.47.1, 00000000004.43.1, 00000000002.60.1, 00000000004.91.1, 00000000002.30.1, 00000000002.65.1, 00000000004.78.1, 00000000002.52.1, 00000000002.81.1, 00000000002.55.1] 2021-02-11T06:44:30.243 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [00000000004.45.1, 00000000002.47.1, 00000000004.43.1, 00000000002.60.1, 00000000004.91.1, 00000000002.30.1, 00000000002.65.1, 00000000004.78.1, 00000000002.52.1, 00000000002.81.1, 00000000002.55.1] 2021-02-11T06:44:30.243 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:44:30.240+0000 7f02a4440080 0 bucket checkpoint complete 2021-02-11T06:44:30.275 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-63', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:44:30.275 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-63', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:44:30.275 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-63 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:44:30.321 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:44:30.321 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:44:30.590 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:44:30.588+0000 7f8e8602f080 1 bucket sync caught up with source: 2021-02-11T06:44:30.590 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [00000000004.45.1, 00000000002.47.1, 00000000004.43.1, 00000000002.60.1, 00000000004.91.1, 00000000002.30.1, 00000000002.65.1, 00000000004.78.1, 00000000002.52.1, 00000000002.81.1, 00000000002.55.1] 2021-02-11T06:44:30.590 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [00000000004.45.1, 00000000002.47.1, 00000000004.43.1, 00000000002.60.1, 00000000004.91.1, 00000000002.30.1, 00000000002.65.1, 00000000004.78.1, 00000000002.52.1, 00000000002.81.1, 00000000002.55.1] 2021-02-11T06:44:30.590 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:44:30.588+0000 7f8e8602f080 0 bucket checkpoint complete 2021-02-11T06:44:30.626 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-63', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:44:30.626 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-63', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:44:30.626 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-63 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:44:30.673 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:44:30.674 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:44:30.964 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:44:30.960+0000 7f0839d6a080 1 bucket sync caught up with source: 2021-02-11T06:44:30.965 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [00000000004.49.5, 00000000002.51.1, 00000000004.35.5, 00000000002.67.1, 00000000004.64.5, 00000000002.33.1, 00000000002.68.1, 00000000004.81.5, 00000000002.47.1, 00000000002.62.1, 00000000002.50.1] 2021-02-11T06:44:30.965 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [00000000004.49.5, 00000000002.51.1, 00000000004.35.5, 00000000002.67.1, 00000000004.64.5, 00000000002.33.1, 00000000002.68.1, 00000000004.81.5, 00000000002.47.1, 00000000002.62.1, 00000000002.50.1] 2021-02-11T06:44:30.965 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:44:30.960+0000 7f0839d6a080 0 bucket checkpoint complete 2021-02-11T06:44:30.988 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-63', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:44:30.988 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-63', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:44:30.988 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-63 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:44:31.038 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:44:31.038 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:44:31.288 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:44:31.283+0000 7f76fb4ca080 1 bucket sync caught up with source: 2021-02-11T06:44:31.288 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [00000000004.49.5, 00000000002.51.1, 00000000004.35.5, 00000000002.67.1, 00000000004.64.5, 00000000002.33.1, 00000000002.68.1, 00000000004.81.5, 00000000002.47.1, 00000000002.62.1, 00000000002.50.1] 2021-02-11T06:44:31.288 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [00000000004.49.5, 00000000002.51.1, 00000000004.35.5, 00000000002.67.1, 00000000004.64.5, 00000000002.33.1, 00000000002.68.1, 00000000004.81.5, 00000000002.47.1, 00000000002.62.1, 00000000002.50.1] 2021-02-11T06:44:31.288 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:44:31.283+0000 7f76fb4ca080 0 bucket checkpoint complete 2021-02-11T06:44:31.319 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-63 zones={test-zone2, test-zone1} 2021-02-11T06:44:31.329 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:44:31.335 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T06:44:31.335 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T06:44:31.335 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T06:44:31.335 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T06:44:31.335 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T06:44:31.335 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T06:44:31.335 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T06:44:31.335 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T06:44:31.335 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:44:31.342 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T06:44:31.342 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T06:44:31.342 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T06:44:31.342 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T06:44:31.342 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T06:44:31.342 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T06:44:31.342 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T06:44:31.342 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T06:44:31.354 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:44:31.407 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:44:31.519 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:44:31.627 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:44:31.680 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:44:31.793 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:44:31.898 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:44:31.952 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:44:32.064 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:44:32.171 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:44:32.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:44:32.334 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:44:32.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T06:44:32.492 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T06:44:32.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T06:44:32.714 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T06:44:32.768 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T06:44:32.880 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T06:44:32.986 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T06:44:33.040 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T06:44:33.152 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T06:44:33.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T06:44:33.311 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T06:44:33.423 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T06:44:33.530 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-63 zones={test-zone2, test-zone1} 2021-02-11T06:44:33.531 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-63 zones={test-zone3, test-zone1} 2021-02-11T06:44:33.540 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:44:33.545 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T06:44:33.545 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T06:44:33.545 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T06:44:33.545 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T06:44:33.545 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T06:44:33.545 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T06:44:33.545 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T06:44:33.545 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T06:44:33.546 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:44:33.551 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T06:44:33.551 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T06:44:33.551 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T06:44:33.551 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T06:44:33.551 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T06:44:33.551 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T06:44:33.551 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T06:44:33.551 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T06:44:33.561 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:44:33.616 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:44:33.729 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:44:33.834 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:44:33.888 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:44:34.057 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:44:45.573 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:44:45.627 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:44:45.739 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:44:45.841 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:44:45.895 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:44:46.004 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:44:46.110 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T06:44:46.164 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T06:44:46.273 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T06:44:46.376 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T06:44:46.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T06:44:46.539 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T06:44:46.641 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T06:44:46.695 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T06:44:46.802 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T06:44:46.905 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T06:44:46.962 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T06:44:47.076 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T06:44:47.181 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-63 zones={test-zone3, test-zone1} 2021-02-11T06:44:47.182 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-63 zones={test-zone3, test-zone2} 2021-02-11T06:44:47.192 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:44:47.198 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T06:44:47.198 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T06:44:47.198 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T06:44:47.198 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T06:44:47.198 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T06:44:47.198 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T06:44:47.198 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T06:44:47.198 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T06:44:47.198 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:44:47.204 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T06:44:47.204 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T06:44:47.204 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T06:44:47.205 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T06:44:47.205 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T06:44:47.205 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T06:44:47.205 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T06:44:47.205 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T06:44:47.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:44:47.267 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:44:47.417 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:44:47.522 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:44:47.576 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:44:47.688 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:44:47.794 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:44:47.848 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:44:47.959 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:44:48.066 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:44:48.120 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:44:48.231 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:44:48.337 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T06:44:48.390 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T06:44:48.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T06:44:48.601 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T06:44:48.659 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T06:44:48.771 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T06:44:48.874 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T06:44:48.936 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T06:44:49.072 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T06:44:57.514 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T06:44:57.568 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T06:44:57.680 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T06:44:57.787 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-63 zones={test-zone3, test-zone2} 2021-02-11T06:44:57.787 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:44:57.787 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:44:57.787 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-64 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:44:57.832 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:44:57.832 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:44:58.093 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:44:58.090+0000 7f2ad37e2080 1 bucket sync caught up with source: 2021-02-11T06:44:58.093 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [00000000006.59.5, 00000000004.51.1, 00000000006.69.5, 00000000004.40.1, 00000000006.45.5, 00000000004.60.1, 00000000004.33.1, 00000000006.49.5, 00000000004.67.1, 00000000004.46.1, 00000000004.47.1] 2021-02-11T06:44:58.093 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [00000000006.59.5, 00000000004.51.1, 00000000006.69.5, 00000000004.40.1, 00000000006.45.5, 00000000004.60.1, 00000000004.33.1, 00000000006.49.5, 00000000004.67.1, 00000000004.46.1, 00000000004.47.1] 2021-02-11T06:44:58.093 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:44:58.090+0000 7f2ad37e2080 0 bucket checkpoint complete 2021-02-11T06:44:58.125 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:44:58.125 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:44:58.126 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-64 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:44:58.175 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:44:58.175 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:44:58.445 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:44:58.439+0000 7f53b9cf6080 1 bucket sync caught up with source: 2021-02-11T06:44:58.445 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [00000000006.59.5, 00000000004.51.1, 00000000006.69.5, 00000000004.40.1, 00000000006.45.5, 00000000004.60.1, 00000000004.33.1, 00000000006.49.5, 00000000004.67.1, 00000000004.46.1, 00000000004.47.1] 2021-02-11T06:44:58.445 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [00000000006.59.5, 00000000004.51.1, 00000000006.69.5, 00000000004.40.1, 00000000006.45.5, 00000000004.60.1, 00000000004.33.1, 00000000006.49.5, 00000000004.67.1, 00000000004.46.1, 00000000004.47.1] 2021-02-11T06:44:58.445 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:44:58.443+0000 7f53b9cf6080 0 bucket checkpoint complete 2021-02-11T06:44:58.472 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:44:58.472 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:44:58.473 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-64 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:44:58.520 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:44:58.520 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:44:58.751 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:44:58.746+0000 7f13a585f080 1 bucket sync caught up with source: 2021-02-11T06:44:58.751 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [00000000006.59.5, 00000000004.51.1, 00000000006.69.5, 00000000004.40.1, 00000000006.45.5, 00000000004.60.1, 00000000004.33.1, 00000000006.49.5, 00000000004.67.1, 00000000004.46.1, 00000000004.47.1] 2021-02-11T06:44:58.751 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [00000000006.59.5, 00000000004.51.1, 00000000006.69.5, 00000000004.40.1, 00000000006.45.5, 00000000004.60.1, 00000000004.33.1, 00000000006.49.5, 00000000004.67.1, 00000000004.46.1, 00000000004.47.1] 2021-02-11T06:44:58.751 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:44:58.746+0000 7f13a585f080 0 bucket checkpoint complete 2021-02-11T06:44:58.781 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-64', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:44:58.781 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-64', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:44:58.781 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-64 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:44:58.830 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:44:58.831 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:44:59.095 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:44:59.091+0000 7f1b51b7a080 1 bucket sync caught up with source: 2021-02-11T06:44:59.095 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [00000000004.70.5, 00000000002.45.1, 00000000004.73.5, 00000000002.39.1, 00000000004.46.5, 00000000002.58.1, 00000000002.33.1, 00000000004.54.5, 00000000002.69.1, 00000000002.48.1, 00000000002.49.1] 2021-02-11T06:44:59.095 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [00000000004.70.5, 00000000002.45.1, 00000000004.73.5, 00000000002.39.1, 00000000004.46.5, 00000000002.58.1, 00000000002.33.1, 00000000004.54.5, 00000000002.69.1, 00000000002.48.1, 00000000002.49.1] 2021-02-11T06:44:59.095 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:44:59.091+0000 7f1b51b7a080 0 bucket checkpoint complete 2021-02-11T06:44:59.130 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-64', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:44:59.130 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-64', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:44:59.130 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-64 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:44:59.179 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:44:59.179 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:44:59.440 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:44:59.435+0000 7f7e501a0080 1 bucket sync caught up with source: 2021-02-11T06:45:25.634 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [00000000004.70.5, 00000000002.45.1, 00000000004.73.5, 00000000002.39.1, 00000000004.46.5, 00000000002.58.1, 00000000002.33.1, 00000000004.54.5, 00000000002.69.1, 00000000002.48.1, 00000000002.49.1] 2021-02-11T06:45:25.634 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [00000000004.70.5, 00000000002.45.1, 00000000004.73.5, 00000000002.39.1, 00000000004.46.5, 00000000002.58.1, 00000000002.33.1, 00000000004.54.5, 00000000002.69.1, 00000000002.48.1, 00000000002.49.1] 2021-02-11T06:45:25.634 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:44:59.435+0000 7f7e501a0080 0 bucket checkpoint complete 2021-02-11T06:45:25.636 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-64', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:45:25.636 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-64', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:45:25.636 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-64 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:45:25.687 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:45:25.687 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:45:25.952 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:45:25.947+0000 7f72b73cb080 1 bucket sync caught up with source: 2021-02-11T06:45:25.952 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [00000000004.64.1, 00000000002.44.1, 00000000004.69.1, 00000000002.37.1, 00000000004.41.1, 00000000002.58.1, 00000000002.36.1, 00000000004.52.1, 00000000002.72.1, 00000000002.53.1, 00000000002.56.1] 2021-02-11T06:45:25.952 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [00000000004.64.1, 00000000002.44.1, 00000000004.69.1, 00000000002.37.1, 00000000004.41.1, 00000000002.58.1, 00000000002.36.1, 00000000004.52.1, 00000000002.72.1, 00000000002.53.1, 00000000002.56.1] 2021-02-11T06:45:25.952 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:45:25.947+0000 7f72b73cb080 0 bucket checkpoint complete 2021-02-11T06:45:25.984 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-64', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:45:25.984 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-64', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:45:25.984 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-64 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:45:26.030 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:45:26.030 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:45:26.289 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:45:26.286+0000 7f43edc40080 1 bucket sync caught up with source: 2021-02-11T06:45:26.289 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [00000000004.64.1, 00000000002.44.1, 00000000004.69.1, 00000000002.37.1, 00000000004.41.1, 00000000002.58.1, 00000000002.36.1, 00000000004.52.1, 00000000002.72.1, 00000000002.53.1, 00000000002.56.1] 2021-02-11T06:45:26.289 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [00000000004.64.1, 00000000002.44.1, 00000000004.69.1, 00000000002.37.1, 00000000004.41.1, 00000000002.58.1, 00000000002.36.1, 00000000004.52.1, 00000000002.72.1, 00000000002.53.1, 00000000002.56.1] 2021-02-11T06:45:26.289 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:45:26.286+0000 7f43edc40080 0 bucket checkpoint complete 2021-02-11T06:45:26.318 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-64 zones={test-zone2, test-zone1} 2021-02-11T06:45:26.331 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:45:26.336 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T06:45:26.336 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T06:45:26.336 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T06:45:26.336 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T06:45:26.337 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T06:45:26.337 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T06:45:26.337 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T06:45:26.337 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T06:45:26.337 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:45:26.343 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T06:45:26.343 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T06:45:26.343 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T06:45:26.343 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T06:45:26.343 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T06:45:26.343 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T06:45:26.343 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T06:45:26.343 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T06:45:26.355 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:45:26.408 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:45:26.521 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:45:26.626 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:45:26.680 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:45:26.792 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:45:26.899 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:45:26.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:45:27.064 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:45:27.175 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:45:27.231 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:45:27.340 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:45:27.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T06:45:27.505 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T06:45:27.615 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T06:45:27.722 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T06:45:27.776 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T06:45:27.894 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T06:45:28.003 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T06:45:28.060 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T06:45:28.169 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T06:45:28.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T06:45:28.327 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T06:45:28.441 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T06:45:28.546 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-64 zones={test-zone2, test-zone1} 2021-02-11T06:45:28.547 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-64 zones={test-zone3, test-zone1} 2021-02-11T06:45:28.556 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:45:28.562 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T06:45:28.562 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T06:45:28.562 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T06:45:28.562 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T06:45:28.562 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T06:45:28.562 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T06:45:28.562 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T06:45:28.562 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T06:45:28.562 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:45:28.568 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T06:45:28.568 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T06:45:28.568 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T06:45:28.568 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T06:45:28.568 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T06:45:28.568 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T06:45:28.568 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T06:45:28.568 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T06:45:28.580 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:45:28.637 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:45:28.750 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:45:28.853 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:45:28.918 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:45:29.552 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:45:29.658 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:45:29.712 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:45:29.824 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:45:29.928 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:45:29.984 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:45:30.096 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:45:30.201 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T06:45:30.256 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T06:45:30.368 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T06:45:30.471 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T06:45:30.528 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T06:45:30.642 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T06:45:30.745 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T06:45:30.801 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T06:45:30.915 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T06:45:31.017 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T06:45:31.074 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T06:45:31.188 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T06:45:31.294 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-64 zones={test-zone3, test-zone1} 2021-02-11T06:45:31.294 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-64 zones={test-zone3, test-zone2} 2021-02-11T06:45:31.305 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:45:31.311 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T06:45:31.311 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T06:45:31.311 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T06:45:31.311 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T06:45:31.311 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T06:45:31.311 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T06:45:31.311 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T06:45:31.311 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T06:45:31.311 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:45:31.317 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T06:45:31.317 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T06:45:31.317 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T06:45:31.317 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T06:45:31.317 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T06:45:31.317 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T06:45:31.318 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T06:45:31.318 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T06:45:31.328 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:45:31.379 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:45:31.488 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T06:45:31.598 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:45:31.652 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:45:31.764 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T06:45:31.869 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:45:31.924 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:45:32.037 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T06:45:32.142 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:45:32.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:45:32.303 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T06:45:32.410 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T06:45:32.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T06:45:32.578 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T06:45:32.682 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T06:45:32.736 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T06:45:32.848 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T06:45:32.958 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T06:45:33.011 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T06:45:33.125 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T06:45:33.234 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T06:45:33.288 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T06:45:33.400 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T06:45:33.505 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-64 zones={test-zone3, test-zone2} 2021-02-11T06:45:33.506 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:45:33.506 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:45:33.506 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:45:33.546 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:45:33.546 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:45:33.838 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"00000000000000000000:00000000000000016983","last_update":"2021-02-11T06:43:37.190003Z"},{"marker":"00000000000000000000:00000000000000014359","last_update":"2021-02-11T06:43:38.304626Z"},{"marker":"00000000000000000000:00000000000000012071","last_update":"2021-02-11T06:43:38.304768Z"},{"marker":"00000000000000000000:00000000000000018712","last_update":"2021-02-11T06:43:38.317125Z"}] 2021-02-11T06:45:50.047 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-02-11T06:45:50.047 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:45:50.047 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:45:50.048 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:45:50.092 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:45:50.092 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:45:50.351 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":15153533235123629405},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000016983","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:43:37.190003Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000014359","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:43:38.304626Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000012071","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:43:38.304768Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000018712","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:43:38.317125Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T06:45:50.351 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-02-11T06:45:50.351 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:45:50.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:45:50.352 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:45:50.390 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:45:50.391 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:45:50.698 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"00000000000000000000:00000000000000016983","last_update":"2021-02-11T06:43:37.190003Z"},{"marker":"00000000000000000000:00000000000000014359","last_update":"2021-02-11T06:43:38.304626Z"},{"marker":"00000000000000000000:00000000000000012071","last_update":"2021-02-11T06:43:38.304768Z"},{"marker":"00000000000000000000:00000000000000018712","last_update":"2021-02-11T06:43:38.317125Z"}] 2021-02-11T06:45:50.698 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-02-11T06:45:50.698 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:45:50.698 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:45:50.698 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:45:50.784 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:45:50.784 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:45:51.053 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":15040254564068855740},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000016983","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:43:37.190003Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000014359","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:43:38.304626Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000012071","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:43:38.304768Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000018712","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:43:38.317125Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T06:45:51.054 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-02-11T06:45:51.054 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:45:51.054 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:45:51.054 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:45:51.135 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:45:51.136 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:45:51.432 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"00000000000000000000:00000000000000016983","last_update":"2021-02-11T06:43:37.190003Z"},{"marker":"00000000000000000000:00000000000000014359","last_update":"2021-02-11T06:43:38.304626Z"},{"marker":"00000000000000000000:00000000000000012071","last_update":"2021-02-11T06:43:38.304768Z"},{"marker":"00000000000000000000:00000000000000018712","last_update":"2021-02-11T06:43:38.317125Z"}] 2021-02-11T06:45:51.433 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-02-11T06:45:51.433 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:45:51.433 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:45:51.433 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:45:51.476 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:45:51.476 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:45:51.733 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":1899655024930128290},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000016983","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:43:37.190003Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000014359","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:43:38.304626Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000012071","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:43:38.304768Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000018712","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:43:38.317125Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T06:45:51.734 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-02-11T06:45:51.734 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:45:51.734 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:45:51.734 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:45:51.816 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:45:51.816 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:45:52.092 INFO:teuthology.orchestra.run.smithi172.stdout:[{"marker":"00000000000000000000:00000000000000008613","last_update":"2021-02-11T06:43:59.279271Z"},{"marker":"00000000000000000000:00000000000000005568","last_update":"2021-02-11T06:43:59.282723Z"},{"marker":"00000000000000000000:00000000000000005959","last_update":"2021-02-11T06:43:59.285466Z"},{"marker":"00000000000000000000:00000000000000009677","last_update":"2021-02-11T06:43:59.288022Z"}] 2021-02-11T06:45:52.093 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-02-11T06:45:52.093 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:45:52.093 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:45:52.093 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:45:52.138 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:45:52.139 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:45:52.418 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":3699056310506273018},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000008613","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:43:59.279271Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005568","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:43:59.282723Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005959","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:43:59.285466Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000009677","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:43:59.288022Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T06:45:52.419 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-02-11T06:45:52.419 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:45:52.419 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:45:52.419 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:45:52.456 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:45:52.456 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:45:52.749 INFO:teuthology.orchestra.run.smithi172.stdout:[{"marker":"00000000000000000000:00000000000000008613","last_update":"2021-02-11T06:43:59.279271Z"},{"marker":"00000000000000000000:00000000000000005568","last_update":"2021-02-11T06:43:59.282723Z"},{"marker":"00000000000000000000:00000000000000005959","last_update":"2021-02-11T06:43:59.285466Z"},{"marker":"00000000000000000000:00000000000000009677","last_update":"2021-02-11T06:43:59.288022Z"}] 2021-02-11T06:45:52.749 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-02-11T06:45:52.749 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:45:52.749 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:45:52.749 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:45:52.792 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:45:52.793 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:45:53.071 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":995053126825359944},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000008613","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:43:59.279271Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005568","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:43:59.282723Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005959","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:43:59.285466Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000009677","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:43:59.288022Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T06:45:53.071 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-02-11T06:45:53.072 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:45:53.072 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:45:53.072 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:45:53.153 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:45:53.153 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:45:53.447 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"00000000000000000000:00000000000000008481","last_update":"2021-02-11T06:43:58.949439Z"},{"marker":"00000000000000000000:00000000000000005701","last_update":"2021-02-11T06:43:58.960494Z"},{"marker":"00000000000000000000:00000000000000006092","last_update":"2021-02-11T06:43:58.963720Z"},{"marker":"00000000000000000000:00000000000000009810","last_update":"2021-02-11T06:43:58.966341Z"}] 2021-02-11T06:45:53.447 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-02-11T06:45:53.448 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:45:53.448 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:45:53.448 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:45:53.531 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:45:53.532 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:45:53.806 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":398085786125260623},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000008481","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:43:58.949439Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005701","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:43:58.960494Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000006092","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:43:58.963720Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000009810","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:43:58.966341Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T06:46:11.497 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-02-11T06:46:11.498 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:46:11.498 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:46:11.498 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:46:11.538 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:46:11.538 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:46:11.829 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"00000000000000000000:00000000000000008481","last_update":"2021-02-11T06:43:58.949439Z"},{"marker":"00000000000000000000:00000000000000005701","last_update":"2021-02-11T06:43:58.960494Z"},{"marker":"00000000000000000000:00000000000000006092","last_update":"2021-02-11T06:43:58.963720Z"},{"marker":"00000000000000000000:00000000000000009810","last_update":"2021-02-11T06:43:58.966341Z"}] 2021-02-11T06:46:11.829 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-02-11T06:46:11.829 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:46:11.829 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:46:11.830 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:46:11.872 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:46:11.872 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:46:12.153 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":11837782300501369896},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000008481","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:43:58.949439Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000005701","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:43:58.960494Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000006092","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:43:58.963720Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000009810","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:43:58.966341Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T06:46:12.154 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-02-11T06:46:12.155 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_sync_disable_enable ... ok 2021-02-11T06:46:12.156 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xscaju-65 2021-02-11T06:46:12.203 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=xscaju-66 2021-02-11T06:46:12.276 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=xscaju-67 2021-02-11T06:46:14.873 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:46:14.873 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:46:14.874 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-65 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:46:14.922 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:46:14.922 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:46:15.158 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:46:15.152+0000 7f18258b5080 1 waiting for incremental sync to catch up: 2021-02-11T06:46:15.158 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , 00000000005.70.5, , , , , ] 2021-02-11T06:46:15.158 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , 00000000011.74.5, , , , , ] 2021-02-11T06:46:20.162 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:46:20.156+0000 7f18258b5080 1 bucket sync caught up with source: 2021-02-11T06:46:20.162 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , 00000000011.74.5, , , , , ] 2021-02-11T06:46:20.162 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , 00000000011.74.5, , , , , ] 2021-02-11T06:46:20.162 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:46:20.156+0000 7f18258b5080 0 bucket checkpoint complete 2021-02-11T06:46:20.192 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:46:20.192 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:46:20.193 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-65 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:46:20.241 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:46:20.241 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:46:20.510 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:46:20.506+0000 7f805a558080 1 bucket sync caught up with source: 2021-02-11T06:46:20.510 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, , , , , 00000000011.74.5, , , , , ] 2021-02-11T06:46:20.511 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, , , , , 00000000011.74.5, , , , , ] 2021-02-11T06:46:20.511 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:46:20.506+0000 7f805a558080 0 bucket checkpoint complete 2021-02-11T06:46:20.542 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:46:20.542 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:46:20.543 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-65 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:46:20.589 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:46:20.589 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:46:20.827 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:46:20.820+0000 7f143ef2c080 1 bucket sync caught up with source: 2021-02-11T06:46:20.827 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , 00000000011.74.5, , , , , ] 2021-02-11T06:46:20.828 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , 00000000011.74.5, , , , , ] 2021-02-11T06:46:20.828 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:46:20.820+0000 7f143ef2c080 0 bucket checkpoint complete 2021-02-11T06:46:20.858 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-65', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:46:20.858 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-65', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:46:20.858 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-65 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:46:20.906 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:46:20.906 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:46:21.198 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:46:21.194+0000 7f4f23bf7080 1 bucket sync caught up with source: 2021-02-11T06:46:21.199 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, , , , , 00000000001.68.5, , , , , ] 2021-02-11T06:46:21.199 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, , , , , 00000000001.68.5, , , , , ] 2021-02-11T06:46:21.199 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:46:21.194+0000 7f4f23bf7080 0 bucket checkpoint complete 2021-02-11T06:46:21.233 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-65', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:46:21.233 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-65', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:46:21.233 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-65 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:46:21.277 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:46:21.277 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:46:21.545 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:46:21.542+0000 7f5242de8080 1 bucket sync caught up with source: 2021-02-11T06:46:34.846 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, , , , , 00000000001.68.5, , , , , ] 2021-02-11T06:46:34.846 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, , , , , 00000000001.68.5, , , , , ] 2021-02-11T06:46:34.846 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:46:21.542+0000 7f5242de8080 0 bucket checkpoint complete 2021-02-11T06:46:34.848 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-65', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:46:34.848 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-65', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:46:34.848 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-65 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:46:34.898 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:46:34.898 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:46:35.167 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:46:35.162+0000 7f0f3a702080 1 bucket sync caught up with source: 2021-02-11T06:46:35.167 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, , , , , 00000000001.78.5, , , , , ] 2021-02-11T06:46:35.167 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, , , , , 00000000001.78.5, , , , , ] 2021-02-11T06:46:35.167 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:46:35.162+0000 7f0f3a702080 0 bucket checkpoint complete 2021-02-11T06:46:35.196 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-65', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:46:35.196 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-65', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:46:35.196 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-65 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:46:35.245 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:46:35.245 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:46:35.484 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:46:35.476+0000 7f96a9d76080 1 bucket sync caught up with source: 2021-02-11T06:46:35.484 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , 00000000001.78.5, , , , , ] 2021-02-11T06:46:35.485 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , 00000000001.78.5, , , , , ] 2021-02-11T06:46:35.485 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:46:35.476+0000 7f96a9d76080 0 bucket checkpoint complete 2021-02-11T06:46:35.518 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-65 zones={test-zone2, test-zone1} 2021-02-11T06:46:35.529 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:46:35.583 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2021-02-11T06:46:35.583 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:46:35.590 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2021-02-11T06:46:35.650 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2021-02-11T06:46:35.903 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2021-02-11T06:46:36.217 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2021-02-11T06:46:36.519 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-65 zones={test-zone2, test-zone1} 2021-02-11T06:46:36.519 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-65 zones={test-zone3, test-zone1} 2021-02-11T06:46:36.531 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:46:36.579 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2021-02-11T06:46:36.579 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:46:36.584 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2021-02-11T06:46:36.637 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2021-02-11T06:46:36.871 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2021-02-11T06:46:37.165 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2021-02-11T06:46:37.439 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-65 zones={test-zone3, test-zone1} 2021-02-11T06:46:37.439 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-65 zones={test-zone3, test-zone2} 2021-02-11T06:46:37.450 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:46:37.499 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2021-02-11T06:46:37.499 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:46:37.507 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2021-02-11T06:46:37.561 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2021-02-11T06:46:37.812 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2021-02-11T06:46:38.120 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2021-02-11T06:46:38.423 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-65 zones={test-zone3, test-zone2} 2021-02-11T06:46:38.423 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_multipart_object_sync ... ok 2021-02-11T06:46:38.423 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xscaju-68 2021-02-11T06:46:38.563 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:46:38.563 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:46:38.563 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:46:38.605 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:46:38.605 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:46:38.622 INFO:tasks.rgw.c2.client.1.smithi172.stdout:2021-02-11T06:46:38.616+0000 7f2bceffd700 -1 data sync zone:5f91a5b1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T06:46:38.622 INFO:tasks.rgw.c2.client.1.smithi172.stdout:2021-02-11T06:46:38.616+0000 7f2bc6ffd700 -1 data sync zone:5f91a5b1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T06:46:38.622 INFO:tasks.rgw.c2.client.0.smithi172.stdout:2021-02-11T06:46:38.616+0000 7fb3c47f8700 -1 data sync zone:5f91a5b1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T06:46:38.622 INFO:tasks.rgw.c2.client.0.smithi172.stdout:2021-02-11T06:46:38.616+0000 7fb3c07f0700 -1 data sync zone:5f91a5b1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T06:46:38.623 INFO:tasks.rgw.c1.client.1.smithi061.stdout:2021-02-11T06:46:38.618+0000 7f701c7f0700 -1 data sync zone:5f91a5b1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T06:46:38.623 INFO:tasks.rgw.c1.client.1.smithi061.stdout:2021-02-11T06:46:38.618+0000 7f702b7fe700 -1 data sync zone:5f91a5b1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T06:46:38.829 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=0d061ff0-ede8-4baa-beef-d975d317d505 2021-02-11T06:46:38.871 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613025998.469131_318.1","last_update":"2021-02-11T06:46:38.469131Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:46:38.871 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T06:46:38.872 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:46:38.872 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:46:38.872 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:46:38.912 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:46:38.913 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:46:39.171 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613025998.469131_318.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:46:38.469131Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:46:39.172 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T06:46:39.172 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T06:46:39.172 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:46:39.172 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:46:39.172 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:46:39.214 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:46:39.214 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:46:39.518 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613025998.469131_318.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:46:38.469131Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:46:39.518 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T06:46:39.519 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:46:39.519 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:46:39.519 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:46:39.519 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:46:39.559 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:46:39.560 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:46:39.817 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613025998.469131_318.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:46:38.469131Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:46:54.964 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:46:54.965 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-68', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:46:54.965 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-68', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:46:54.965 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-68 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:46:55.012 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:46:55.013 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:46:55.279 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:46:55.271+0000 7f3e65192080 1 bucket sync caught up with source: 2021-02-11T06:46:55.280 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000001.53.5, , , , 00000000001.36.5, , , , , ] 2021-02-11T06:46:55.280 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000001.53.5, , , , 00000000001.36.5, , , , , ] 2021-02-11T06:46:55.280 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:46:55.271+0000 7f3e65192080 0 bucket checkpoint complete 2021-02-11T06:46:55.418 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_encrypted_object_sync ... ok 2021-02-11T06:46:55.419 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xscaju-69 2021-02-11T06:46:55.631 INFO:tasks.rgw.c2.client.0.smithi172.stdout:2021-02-11T06:46:55.623+0000 7fb3ccff9700 -1 data sync zone:5f91a5b1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T06:46:55.631 INFO:tasks.rgw.c2.client.0.smithi172.stdout:2021-02-11T06:46:55.623+0000 7fb3ceffd700 -1 data sync zone:5f91a5b1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T06:46:55.631 INFO:tasks.rgw.c2.client.0.smithi172.stdout:2021-02-11T06:46:55.623+0000 7fb3cffff700 -1 data sync zone:5f91a5b1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T06:46:55.631 INFO:tasks.rgw.c2.client.0.smithi172.stdout:2021-02-11T06:46:55.623+0000 7fb3c17f2700 -1 data sync zone:5f91a5b1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T06:46:55.631 INFO:tasks.rgw.c2.client.1.smithi172.stdout:2021-02-11T06:46:55.623+0000 7f2bc07f0700 -1 data sync zone:5f91a5b1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T06:46:55.632 INFO:tasks.rgw.c2.client.1.smithi172.stdout:2021-02-11T06:46:55.623+0000 7f2bcdffb700 -1 data sync zone:5f91a5b1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T06:46:55.632 INFO:tasks.rgw.c2.client.1.smithi172.stdout:2021-02-11T06:46:55.623+0000 7f2bc27f4700 -1 data sync zone:5f91a5b1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T06:46:55.632 INFO:tasks.rgw.c1.client.1.smithi061.stdout:2021-02-11T06:46:55.629+0000 7f701a7ec700 -1 data sync zone:5f91a5b1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T06:46:55.632 INFO:tasks.rgw.c1.client.1.smithi061.stdout:2021-02-11T06:46:55.629+0000 7f701c7f0700 -1 data sync zone:5f91a5b1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T06:46:55.632 INFO:tasks.rgw.c2.client.1.smithi172.stdout:2021-02-11T06:46:55.623+0000 7f2bc77fe700 -1 data sync zone:5f91a5b1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T06:46:55.633 INFO:tasks.rgw.c1.client.1.smithi061.stdout:2021-02-11T06:46:55.629+0000 7f701afed700 -1 data sync zone:5f91a5b1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T06:46:55.633 INFO:tasks.rgw.c1.client.1.smithi061.stdout:2021-02-11T06:46:55.633+0000 7f701bfef700 -1 data sync zone:5f91a5b1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T06:46:55.659 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:46:55.659 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:46:55.659 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:46:55.703 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:46:55.703 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:46:55.918 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=0d061ff0-ede8-4baa-beef-d975d317d505 2021-02-11T06:46:55.967 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613026015.463560_322.1","last_update":"2021-02-11T06:46:55.463560Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:46:55.968 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T06:46:55.968 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:46:55.968 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:46:55.968 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:46:56.008 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:46:56.009 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:46:56.272 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026015.463560_322.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:46:55.463560Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:46:56.272 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T06:46:56.272 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T06:46:56.273 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:46:56.273 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:46:56.273 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:46:56.314 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:46:56.314 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:46:56.595 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026015.463560_322.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:46:55.463560Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:46:56.595 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T06:46:56.595 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:46:56.596 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:46:56.596 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:46:56.596 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:46:56.637 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:46:56.637 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:46:56.910 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026015.463560_322.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:46:55.463560Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:46:56.911 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:46:56.911 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-69', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:46:56.911 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-69', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:46:56.911 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-69 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:46:57.000 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:46:57.000 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:46:57.260 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:46:57.251+0000 7fc2272fc080 1 bucket sync caught up with source: 2021-02-11T06:46:57.260 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000001.67.5, , , , 00000000001.121.5, , , 00000000001.69.5, , 00000000001.63.5] 2021-02-11T06:46:57.260 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000001.67.5, , , , 00000000001.121.5, , , 00000000001.69.5, , 00000000001.63.5] 2021-02-11T06:46:57.261 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:46:57.251+0000 7fc2272fc080 0 bucket checkpoint complete 2021-02-11T06:46:57.292 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-69', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:46:57.292 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-69', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:46:57.293 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-69 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:46:57.341 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:46:57.341 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:46:57.603 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:46:57.601+0000 7f31f1cc6080 1 bucket sync caught up with source: 2021-02-11T06:46:57.603 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, 00000000001.67.5, , , , 00000000001.121.5, , , 00000000001.69.5, , 00000000001.63.5] 2021-02-11T06:46:57.603 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, 00000000001.67.5, , , , 00000000001.121.5, , , 00000000001.69.5, , 00000000001.63.5] 2021-02-11T06:46:57.603 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:46:57.601+0000 7f31f1cc6080 0 bucket checkpoint complete 2021-02-11T06:46:57.632 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-69', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:46:57.632 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-69', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:46:57.632 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-69 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:46:57.680 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:46:57.681 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:46:57.925 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:46:57.919+0000 7fe6b3bda080 1 bucket sync caught up with source: 2021-02-11T06:46:57.926 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000001.67.5, , , , 00000000001.121.5, , , 00000000001.69.5, , 00000000001.63.5] 2021-02-11T06:46:57.926 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000001.67.5, , , , 00000000001.121.5, , , 00000000001.69.5, , 00000000001.63.5] 2021-02-11T06:46:57.926 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:46:57.919+0000 7fe6b3bda080 0 bucket checkpoint complete 2021-02-11T06:46:57.957 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-69', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:46:57.957 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-69', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:46:57.957 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-69 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:46:58.007 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:46:58.007 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:46:58.273 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:46:58.273+0000 7f5a7d5f2080 1 bucket sync caught up with source: 2021-02-11T06:46:58.273 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, 00000000001.71.5, , , , 00000000001.86.5, , , 00000000001.68.5, , 00000000001.79.5] 2021-02-11T06:46:58.273 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, 00000000001.71.5, , , , 00000000001.86.5, , , 00000000001.68.5, , 00000000001.79.5] 2021-02-11T06:46:58.273 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:46:58.273+0000 7f5a7d5f2080 0 bucket checkpoint complete 2021-02-11T06:46:58.309 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-69', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:46:58.309 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-69', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:46:58.310 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-69 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:46:58.357 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:46:58.357 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:46:58.632 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:46:58.629+0000 7f33735d6080 1 bucket sync caught up with source: 2021-02-11T06:46:58.632 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, 00000000001.71.5, , , , 00000000001.86.5, , , 00000000001.68.5, , 00000000001.79.5] 2021-02-11T06:46:58.632 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, 00000000001.71.5, , , , 00000000001.86.5, , , 00000000001.68.5, , 00000000001.79.5] 2021-02-11T06:46:58.632 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:46:58.629+0000 7f33735d6080 0 bucket checkpoint complete 2021-02-11T06:46:58.667 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-69', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:46:58.667 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-69', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:46:58.667 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-69 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:46:58.714 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:46:58.714 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:46:58.983 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:46:58.981+0000 7f25f2d67080 1 bucket sync caught up with source: 2021-02-11T06:46:58.983 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, 00000000001.66.5, , , , 00000000001.88.5, , , 00000000001.74.5, , 00000000001.69.5] 2021-02-11T06:46:58.983 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, 00000000001.66.5, , , , 00000000001.88.5, , , 00000000001.74.5, , 00000000001.69.5] 2021-02-11T06:46:58.983 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:46:58.981+0000 7f25f2d67080 0 bucket checkpoint complete 2021-02-11T06:46:59.018 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-69', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:46:59.018 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-69', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:46:59.018 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-69 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:46:59.066 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:46:59.066 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:46:59.303 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:46:59.295+0000 7f1281640080 1 bucket sync caught up with source: 2021-02-11T06:46:59.303 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000001.66.5, , , , 00000000001.88.5, , , 00000000001.74.5, , 00000000001.69.5] 2021-02-11T06:46:59.303 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000001.66.5, , , , 00000000001.88.5, , , 00000000001.74.5, , 00000000001.69.5] 2021-02-11T06:46:59.303 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:46:59.295+0000 7f1281640080 0 bucket checkpoint complete 2021-02-11T06:46:59.337 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-69 zones={test-zone2, test-zone1} 2021-02-11T06:46:59.348 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:46:59.396 DEBUG:tasks.rgw_multi.zone_rados:o=a 2021-02-11T06:46:59.396 DEBUG:tasks.rgw_multi.zone_rados:o=b 2021-02-11T06:46:59.396 DEBUG:tasks.rgw_multi.zone_rados:o=c 2021-02-11T06:46:59.396 DEBUG:tasks.rgw_multi.zone_rados:o=d 2021-02-11T06:46:59.396 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:46:59.405 DEBUG:tasks.rgw_multi.zone_rados:o=a 2021-02-11T06:46:59.405 DEBUG:tasks.rgw_multi.zone_rados:o=b 2021-02-11T06:46:59.405 DEBUG:tasks.rgw_multi.zone_rados:o=c 2021-02-11T06:46:59.405 DEBUG:tasks.rgw_multi.zone_rados:o=d 2021-02-11T06:46:59.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-02-11T06:46:59.482 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-02-11T06:47:00.127 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-02-11T06:47:00.230 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-02-11T06:47:00.282 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-02-11T06:47:00.392 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-02-11T06:47:00.498 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-02-11T06:47:00.552 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-02-11T06:47:00.665 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-02-11T06:47:00.770 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-02-11T06:47:00.824 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-02-11T06:47:00.935 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-02-11T06:47:01.038 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-69 zones={test-zone2, test-zone1} 2021-02-11T06:47:01.038 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-69 zones={test-zone3, test-zone1} 2021-02-11T06:47:01.047 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:47:01.051 DEBUG:tasks.rgw_multi.zone_rados:o=a 2021-02-11T06:47:01.051 DEBUG:tasks.rgw_multi.zone_rados:o=b 2021-02-11T06:47:01.052 DEBUG:tasks.rgw_multi.zone_rados:o=c 2021-02-11T06:47:01.052 DEBUG:tasks.rgw_multi.zone_rados:o=d 2021-02-11T06:47:01.052 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:47:01.056 DEBUG:tasks.rgw_multi.zone_rados:o=a 2021-02-11T06:47:01.056 DEBUG:tasks.rgw_multi.zone_rados:o=b 2021-02-11T06:47:01.056 DEBUG:tasks.rgw_multi.zone_rados:o=c 2021-02-11T06:47:01.056 DEBUG:tasks.rgw_multi.zone_rados:o=d 2021-02-11T06:47:01.066 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-02-11T06:47:01.119 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-02-11T06:47:01.228 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-02-11T06:47:01.333 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-02-11T06:47:01.385 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-02-11T06:47:01.496 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-02-11T06:47:01.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-02-11T06:47:01.657 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-02-11T06:47:01.766 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-02-11T06:47:01.869 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-02-11T06:47:01.924 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-02-11T06:47:02.040 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-02-11T06:47:02.150 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-69 zones={test-zone3, test-zone1} 2021-02-11T06:47:21.659 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-69 zones={test-zone3, test-zone2} 2021-02-11T06:47:21.672 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:47:21.677 DEBUG:tasks.rgw_multi.zone_rados:o=a 2021-02-11T06:47:21.677 DEBUG:tasks.rgw_multi.zone_rados:o=b 2021-02-11T06:47:21.678 DEBUG:tasks.rgw_multi.zone_rados:o=c 2021-02-11T06:47:21.678 DEBUG:tasks.rgw_multi.zone_rados:o=d 2021-02-11T06:47:21.678 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:47:21.727 DEBUG:tasks.rgw_multi.zone_rados:o=a 2021-02-11T06:47:21.727 DEBUG:tasks.rgw_multi.zone_rados:o=b 2021-02-11T06:47:21.727 DEBUG:tasks.rgw_multi.zone_rados:o=c 2021-02-11T06:47:21.727 DEBUG:tasks.rgw_multi.zone_rados:o=d 2021-02-11T06:47:21.783 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-02-11T06:47:21.837 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-02-11T06:47:21.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-02-11T06:47:22.054 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-02-11T06:47:22.108 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-02-11T06:47:22.220 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-02-11T06:47:22.324 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-02-11T06:47:22.380 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-02-11T06:47:22.491 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-02-11T06:47:22.594 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-02-11T06:47:22.648 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-02-11T06:47:22.758 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-02-11T06:47:22.866 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-69 zones={test-zone3, test-zone2} 2021-02-11T06:47:22.866 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:47:22.866 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:47:22.867 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 0 --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T06:47:22.904 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:47:22.904 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:47:23.211 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=xscaju-70 2021-02-11T06:47:23.444 INFO:tasks.rgw.c2.client.1.smithi172.stdout:2021-02-11T06:47:23.439+0000 7f2bbd7ea700 -1 data sync zone:5f91a5b1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T06:47:23.444 INFO:tasks.rgw.c2.client.1.smithi172.stdout:2021-02-11T06:47:23.439+0000 7f2bcffff700 -1 data sync zone:5f91a5b1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T06:47:23.445 INFO:tasks.rgw.c2.client.1.smithi172.stdout:2021-02-11T06:47:23.439+0000 7f2bc37f6700 -1 data sync zone:5f91a5b1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T06:47:23.445 INFO:tasks.rgw.c2.client.1.smithi172.stdout:2021-02-11T06:47:23.439+0000 7f2bcd7fa700 -1 data sync zone:5f91a5b1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T06:47:23.446 INFO:tasks.rgw.c1.client.1.smithi061.stdout:2021-02-11T06:47:23.441+0000 7f701dff3700 -1 data sync zone:5f91a5b1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T06:47:23.446 INFO:tasks.rgw.c1.client.1.smithi061.stdout:2021-02-11T06:47:23.445+0000 7f70217fa700 -1 data sync zone:5f91a5b1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T06:47:23.446 INFO:tasks.rgw.c1.client.1.smithi061.stdout:2021-02-11T06:47:23.445+0000 7f702affd700 -1 data sync zone:5f91a5b1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T06:47:23.446 INFO:tasks.rgw.c1.client.1.smithi061.stdout:2021-02-11T06:47:23.445+0000 7f70227fc700 -1 data sync zone:5f91a5b1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T06:47:23.446 INFO:tasks.rgw.c2.client.0.smithi172.stdout:2021-02-11T06:47:23.443+0000 7fb3c67fc700 -1 data sync zone:5f91a5b1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T06:47:23.446 INFO:tasks.rgw.c2.client.0.smithi172.stdout:2021-02-11T06:47:23.443+0000 7fb3c1ff3700 -1 data sync zone:5f91a5b1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T06:47:23.446 INFO:tasks.rgw.c2.client.0.smithi172.stdout:2021-02-11T06:47:23.443+0000 7fb3c37f6700 -1 data sync zone:5f91a5b1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T06:47:23.446 INFO:tasks.rgw.c2.client.0.smithi172.stdout:2021-02-11T06:47:23.443+0000 7fb3bffef700 -1 data sync zone:5f91a5b1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T06:47:23.447 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:47:23.447 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:47:23.447 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:47:23.495 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:47:23.495 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:47:23.722 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=0d061ff0-ede8-4baa-beef-d975d317d505 2021-02-11T06:47:23.753 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613026043.246208_326.1","last_update":"2021-02-11T06:47:23.246208Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:47:23.754 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T06:47:23.754 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:47:23.754 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:47:23.754 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:47:23.796 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:47:23.796 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:47:24.071 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026043.246208_326.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:47:23.246208Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:47:24.072 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T06:47:24.072 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T06:47:24.072 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:47:24.072 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:47:24.072 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:47:24.114 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:47:24.114 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:47:24.414 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026043.246208_326.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:47:23.246208Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:47:24.414 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T06:47:24.414 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:47:24.414 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:47:24.414 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:47:24.415 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:47:24.454 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:47:24.454 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:47:24.720 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026043.246208_326.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:47:23.246208Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:47:37.330 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:47:37.331 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-70', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:47:37.331 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-70', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:47:37.331 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-70 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:47:37.376 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:47:37.376 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:47:37.636 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:47:37.630+0000 7fb40772e080 1 bucket sync caught up with source: 2021-02-11T06:47:37.637 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000001.64.5, , , , 00000000001.73.5, , , 00000000001.57.5, , 00000000001.71.5] 2021-02-11T06:47:37.637 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000001.64.5, , , , 00000000001.73.5, , , 00000000001.57.5, , 00000000001.71.5] 2021-02-11T06:47:37.637 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:47:37.630+0000 7fb40772e080 0 bucket checkpoint complete 2021-02-11T06:47:37.668 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-70', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:47:37.668 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-70', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:47:37.669 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-70 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:47:37.717 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:47:37.717 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:47:37.973 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:47:37.973+0000 7ffa93c4a080 1 bucket sync caught up with source: 2021-02-11T06:47:37.973 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, 00000000001.64.5, , , , 00000000001.73.5, , , 00000000001.57.5, , 00000000001.71.5] 2021-02-11T06:47:37.973 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, 00000000001.64.5, , , , 00000000001.73.5, , , 00000000001.57.5, , 00000000001.71.5] 2021-02-11T06:47:37.973 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:47:37.973+0000 7ffa93c4a080 0 bucket checkpoint complete 2021-02-11T06:47:38.001 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-70', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:47:38.001 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-70', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:47:38.001 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-70 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:47:38.050 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:47:38.050 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:47:38.283 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:47:38.278+0000 7f72e5d2f080 1 bucket sync caught up with source: 2021-02-11T06:47:38.283 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000001.64.5, , , , 00000000001.73.5, , , 00000000001.57.5, , 00000000001.71.5] 2021-02-11T06:47:38.283 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000001.64.5, , , , 00000000001.73.5, , , 00000000001.57.5, , 00000000001.71.5] 2021-02-11T06:47:38.283 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:47:38.278+0000 7f72e5d2f080 0 bucket checkpoint complete 2021-02-11T06:47:38.303 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-70', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:47:38.303 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-70', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:47:38.304 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-70 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:47:38.352 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:47:38.352 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:47:38.630 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:47:38.629+0000 7f7e32dea080 1 bucket sync caught up with source: 2021-02-11T06:47:38.630 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, 00000000001.79.5, , , , 00000000001.80.5, , , 00000000001.72.5, , 00000000001.80.5] 2021-02-11T06:47:38.631 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, 00000000001.79.5, , , , 00000000001.80.5, , , 00000000001.72.5, , 00000000001.80.5] 2021-02-11T06:47:38.631 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:47:38.629+0000 7f7e32dea080 0 bucket checkpoint complete 2021-02-11T06:47:38.666 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-70', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:47:38.666 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-70', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:47:38.666 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-70 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:47:38.713 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:47:38.713 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:47:38.979 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:47:38.972+0000 7f1d34c6a080 1 bucket sync caught up with source: 2021-02-11T06:47:38.980 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, 00000000001.79.5, , , , 00000000001.80.5, , , 00000000001.72.5, , 00000000001.80.5] 2021-02-11T06:47:38.980 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, 00000000001.79.5, , , , 00000000001.80.5, , , 00000000001.72.5, , 00000000001.80.5] 2021-02-11T06:47:38.980 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:47:38.972+0000 7f1d34c6a080 0 bucket checkpoint complete 2021-02-11T06:47:39.011 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-70', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:47:39.011 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-70', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:47:39.011 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-70 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:47:39.064 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:47:39.064 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:47:39.338 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:47:39.336+0000 7f8de0957080 1 bucket sync caught up with source: 2021-02-11T06:47:53.518 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, 00000000001.75.5, , , , 00000000001.86.5, , , 00000000001.72.5, , 00000000001.78.5] 2021-02-11T06:47:53.518 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, 00000000001.75.5, , , , 00000000001.86.5, , , 00000000001.72.5, , 00000000001.78.5] 2021-02-11T06:47:53.519 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:47:39.336+0000 7f8de0957080 0 bucket checkpoint complete 2021-02-11T06:47:53.520 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-70', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:47:53.520 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-70', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:47:53.521 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-70 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:47:53.573 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:47:53.573 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:47:53.829 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:47:53.822+0000 7f787a95a080 1 bucket sync caught up with source: 2021-02-11T06:47:53.829 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000001.75.5, , , , 00000000001.86.5, , , 00000000001.72.5, , 00000000001.78.5] 2021-02-11T06:47:53.830 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000001.75.5, , , , 00000000001.86.5, , , 00000000001.72.5, , 00000000001.78.5] 2021-02-11T06:47:53.831 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:47:53.822+0000 7f787a95a080 0 bucket checkpoint complete 2021-02-11T06:47:53.865 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-70 zones={test-zone2, test-zone1} 2021-02-11T06:47:53.877 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:47:53.883 DEBUG:tasks.rgw_multi.zone_rados:o=a 2021-02-11T06:47:53.884 DEBUG:tasks.rgw_multi.zone_rados:o=b 2021-02-11T06:47:53.884 DEBUG:tasks.rgw_multi.zone_rados:o=c 2021-02-11T06:47:53.884 DEBUG:tasks.rgw_multi.zone_rados:o=d 2021-02-11T06:47:53.884 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:47:53.930 DEBUG:tasks.rgw_multi.zone_rados:o=a 2021-02-11T06:47:53.931 DEBUG:tasks.rgw_multi.zone_rados:o=b 2021-02-11T06:47:53.931 DEBUG:tasks.rgw_multi.zone_rados:o=c 2021-02-11T06:47:53.931 DEBUG:tasks.rgw_multi.zone_rados:o=d 2021-02-11T06:47:53.987 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-02-11T06:47:54.040 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-02-11T06:47:54.157 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-02-11T06:47:54.294 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-02-11T06:47:54.368 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-02-11T06:47:54.487 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-02-11T06:47:54.590 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-02-11T06:47:54.644 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-02-11T06:47:54.755 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-02-11T06:47:54.863 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-02-11T06:47:54.920 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-02-11T06:47:55.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-02-11T06:47:55.142 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-70 zones={test-zone2, test-zone1} 2021-02-11T06:47:55.143 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-70 zones={test-zone3, test-zone1} 2021-02-11T06:47:55.151 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:47:55.156 DEBUG:tasks.rgw_multi.zone_rados:o=a 2021-02-11T06:47:55.156 DEBUG:tasks.rgw_multi.zone_rados:o=b 2021-02-11T06:47:55.156 DEBUG:tasks.rgw_multi.zone_rados:o=c 2021-02-11T06:47:55.156 DEBUG:tasks.rgw_multi.zone_rados:o=d 2021-02-11T06:47:55.156 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:47:55.160 DEBUG:tasks.rgw_multi.zone_rados:o=a 2021-02-11T06:47:55.160 DEBUG:tasks.rgw_multi.zone_rados:o=b 2021-02-11T06:47:55.160 DEBUG:tasks.rgw_multi.zone_rados:o=c 2021-02-11T06:47:55.160 DEBUG:tasks.rgw_multi.zone_rados:o=d 2021-02-11T06:47:55.168 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-02-11T06:47:55.228 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-02-11T06:47:55.343 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-02-11T06:47:55.444 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-02-11T06:47:55.504 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-02-11T06:47:55.615 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-02-11T06:47:55.718 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-02-11T06:47:55.771 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-02-11T06:47:55.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-02-11T06:47:55.985 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-02-11T06:47:56.040 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-02-11T06:47:56.152 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-02-11T06:47:56.259 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-70 zones={test-zone3, test-zone1} 2021-02-11T06:47:56.259 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-70 zones={test-zone3, test-zone2} 2021-02-11T06:47:56.269 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:47:56.319 DEBUG:tasks.rgw_multi.zone_rados:o=a 2021-02-11T06:47:56.319 DEBUG:tasks.rgw_multi.zone_rados:o=b 2021-02-11T06:47:56.319 DEBUG:tasks.rgw_multi.zone_rados:o=c 2021-02-11T06:47:56.319 DEBUG:tasks.rgw_multi.zone_rados:o=d 2021-02-11T06:47:56.319 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:47:56.326 DEBUG:tasks.rgw_multi.zone_rados:o=a 2021-02-11T06:47:56.326 DEBUG:tasks.rgw_multi.zone_rados:o=b 2021-02-11T06:47:56.326 DEBUG:tasks.rgw_multi.zone_rados:o=c 2021-02-11T06:47:56.326 DEBUG:tasks.rgw_multi.zone_rados:o=d 2021-02-11T06:47:56.383 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-02-11T06:47:56.435 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-02-11T06:47:56.552 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-02-11T06:47:56.662 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-02-11T06:47:56.720 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-02-11T06:47:56.832 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-02-11T06:47:56.942 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-02-11T06:47:56.995 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-02-11T06:47:57.103 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-02-11T06:47:57.210 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-02-11T06:47:57.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-02-11T06:47:57.370 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-02-11T06:47:57.474 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-70 zones={test-zone3, test-zone2} 2021-02-11T06:47:57.475 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '1', '--rgw-sync-log-trim-min-cold-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:47:57.475 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '1', '--rgw-sync-log-trim-min-cold-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:47:57.475 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 1 --rgw-sync-log-trim-min-cold-buckets 0 --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T06:47:57.513 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:47:57.513 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:47:57.827 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'xscaju-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:47:57.827 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'xscaju-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:47:57.827 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket xscaju-70 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:47:57.908 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:47:57.908 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:47:58.206 INFO:teuthology.orchestra.run.smithi061.stdout:[] 2021-02-11T06:47:58.207 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'xscaju-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:47:58.207 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'xscaju-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:47:58.207 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket xscaju-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:47:58.287 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:47:58.287 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:47:58.588 INFO:teuthology.orchestra.run.smithi061.stdout:[{"op_id":"1#00000000001.67.5","op_tag":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5509.5768","op":"write","object":"a","instance":"","state":"complete","index_ver":1,"timestamp":"2021-02-11T06:46:55.476572285Z","ver":{"pool":9,"epoch":1},"bilog_flags":0,"versioned":false,"owner":"","owner_display_name":"","zones_trace":[{"entry":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-69:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.20"}]},{"op_id":"5#00000000001.121.5","op_tag":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5509.5783","op":"write","object":"d","instance":"","state":"complete","index_ver":1,"timestamp":"2021-02-11T06:46:55.618286926Z","ver":{"pool":9,"epoch":1},"bilog_flags":0,"versioned":false,"owner":"","owner_display_name":"","zones_trace":[{"entry":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-69:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.20"}]},{"op_id":"8#00000000001.69.5","op_tag":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5509.5772","op":"write","object":"c","instance":"","state":"complete","index_ver":1,"timestamp":"2021-02-11T06:46:55.570868653Z","ver":{"pool":9,"epoch":2},"bilog_flags":0,"versioned":false,"owner":"","owner_display_name":"","zones_trace":[{"entry":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-69:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.20"}]},{"op_id":"10#00000000001.63.5","op_tag":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5509.5770","op":"write","object":"b","instance":"","state":"complete","index_ver":1,"timestamp":"2021-02-11T06:46:55.522906256Z","ver":{"pool":9,"epoch":2},"bilog_flags":0,"versioned":false,"owner":"","owner_display_name":"","zones_trace":[{"entry":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7:xscaju-69:5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.20"}]}] 2021-02-11T06:47:58.588 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '999', '--rgw-sync-log-trim-min-cold-buckets', '999', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:47:58.588 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '999', '--rgw-sync-log-trim-min-cold-buckets', '999', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:47:58.589 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 999 --rgw-sync-log-trim-min-cold-buckets 999 --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T06:47:58.665 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:47:58.665 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:48:00.136 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'xscaju-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:48:10.297 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'xscaju-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:48:10.297 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket xscaju-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:48:10.337 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:48:10.337 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:48:10.622 INFO:teuthology.orchestra.run.smithi061.stdout:[] 2021-02-11T06:48:10.622 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_index_log_trim ... ok 2021-02-11T06:48:10.663 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_creation_time ... ok 2021-02-11T06:48:10.664 INFO:tasks.rgw_multisite_tests: 2021-02-11T06:48:10.664 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2021-02-11T06:48:10.664 INFO:tasks.rgw_multisite_tests:Ran 28 tests in 1832.351s 2021-02-11T06:48:10.664 INFO:tasks.rgw_multisite_tests: 2021-02-11T06:48:10.664 INFO:tasks.rgw_multisite_tests:OK 2021-02-11T06:48:10.670 INFO:tasks.rgw_multisite_tests:log information for manual testing ... ok 2021-02-11T06:48:10.671 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:48:10.671 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:48:10.671 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:48:10.710 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:48:10.710 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:48:10.966 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=0d061ff0-ede8-4baa-beef-d975d317d505 2021-02-11T06:48:11.004 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613026043.246208_326.1","last_update":"2021-02-11T06:47:23.246208Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:48:11.004 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T06:48:11.004 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:48:11.004 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:48:11.005 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:48:11.048 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:48:11.048 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:48:11.310 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026043.246208_326.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:47:23.246208Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:48:11.310 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T06:48:11.310 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T06:48:11.310 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:48:11.311 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:48:11.311 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:48:11.351 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:48:11.351 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:48:11.641 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026043.246208_326.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:47:23.246208Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:48:11.642 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T06:48:11.642 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:48:11.642 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:48:11.642 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:48:11.642 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:48:11.683 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:48:11.684 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:48:11.951 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026043.246208_326.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:47:23.246208Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:48:11.951 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:48:11.951 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:48:11.952 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:48:11.952 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:48:11.992 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:48:11.992 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:48:12.240 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=0d061ff0-ede8-4baa-beef-d975d317d505 2021-02-11T06:48:12.280 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613026043.246208_326.1","last_update":"2021-02-11T06:47:23.246208Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:48:12.280 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:48:12.280 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:48:12.280 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:48:12.281 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:48:12.320 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:48:12.320 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:48:12.587 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026043.246208_326.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:47:23.246208Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:48:12.588 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:48:12.636 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:48:12.636 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:48:12.636 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:48:12.671 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:48:12.672 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:48:12.896 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=0d061ff0-ede8-4baa-beef-d975d317d505 2021-02-11T06:48:12.940 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613026092.631086_333.1","last_update":"2021-02-11T06:48:12.631086Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:48:12.940 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:48:12.940 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:48:12.940 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:48:12.940 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:48:12.979 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:48:12.979 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:48:13.241 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026092.631086_333.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:48:12.631086Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:48:13.241 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:48:13.505 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:48:13.505 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:48:13.505 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:48:13.546 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:48:13.547 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:48:13.795 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=0d061ff0-ede8-4baa-beef-d975d317d505 2021-02-11T06:48:13.837 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613026092.631086_333.1","last_update":"2021-02-11T06:48:12.631086Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:48:13.837 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:48:13.837 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:48:13.837 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:48:13.838 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:48:13.877 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:48:13.877 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:48:14.140 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026092.631086_333.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:48:12.631086Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:48:14.140 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:48:14.283 INFO:tasks.rgw_multisite_tests:test low level implementation of s3 notifications ... ok 2021-02-11T06:48:14.283 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:48:14.283 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:48:14.284 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:48:14.321 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:48:14.321 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:48:14.559 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=0d061ff0-ede8-4baa-beef-d975d317d505 2021-02-11T06:48:14.596 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613026094.269433_335.1","last_update":"2021-02-11T06:48:14.269433Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:48:28.847 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T06:48:28.848 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:48:28.848 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:48:28.848 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:48:28.889 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:48:28.889 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:48:29.182 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026094.269433_335.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:48:14.269433Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:48:29.182 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T06:48:29.182 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T06:48:29.182 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:48:29.183 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:48:29.183 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:48:29.221 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:48:29.221 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:48:29.492 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026094.269433_335.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:48:14.269433Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:48:29.492 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T06:48:29.492 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:48:29.492 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:48:29.492 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:48:29.493 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:48:29.539 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:48:29.539 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:48:29.810 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026094.269433_335.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:48:14.269433Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:48:29.810 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:48:29.810 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:48:29.810 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:48:29.811 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:48:29.858 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:48:29.858 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:48:30.107 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=0d061ff0-ede8-4baa-beef-d975d317d505 2021-02-11T06:48:30.143 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613026094.269433_335.1","last_update":"2021-02-11T06:48:14.269433Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:48:30.143 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:48:30.144 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:48:30.144 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:48:30.144 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:48:30.185 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:48:30.185 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:48:30.455 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026094.269433_335.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:48:14.269433Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:48:30.456 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:48:30.514 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:48:30.514 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:48:30.514 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:48:30.552 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:48:30.552 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:48:30.805 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=0d061ff0-ede8-4baa-beef-d975d317d505 2021-02-11T06:48:30.841 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613026110.508318_339.1","last_update":"2021-02-11T06:48:30.508318Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:48:30.842 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:48:30.842 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:48:30.842 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:48:30.842 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:48:30.882 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:48:30.882 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:48:31.140 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026110.508318_339.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:48:30.508318Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:48:31.141 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:48:31.180 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:48:31.180 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:48:31.180 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:48:31.221 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:48:31.221 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:48:31.469 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=0d061ff0-ede8-4baa-beef-d975d317d505 2021-02-11T06:48:31.508 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613026110.508318_339.1","last_update":"2021-02-11T06:48:30.508318Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:48:31.508 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:48:31.508 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:48:31.508 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:48:31.509 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:48:31.550 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:48:31.550 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:48:31.806 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026110.508318_339.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:48:30.508318Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:48:31.806 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:48:32.227 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-72', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:48:32.227 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-72', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:48:32.227 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-72 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:48:32.273 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:48:32.274 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:48:32.517 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:48:32.513+0000 7fc3e8bf7080 1 waiting for incremental sync to catch up: 2021-02-11T06:48:32.517 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000002.94.5, , 00000000001.95.5, 00000000001.111.5, 00000000001.71.5, , , 00000000001.69.5, 00000000001.109.5, 00000000002.89.5] 2021-02-11T06:48:32.518 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000002.94.5, , 00000000001.95.5, 00000000001.111.5, 00000000001.71.5, , 00000000001.85.5, 00000000001.69.5, 00000000001.109.5, 00000000002.89.5] 2021-02-11T06:48:37.523 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:48:37.517+0000 7fc3e8bf7080 1 waiting for incremental sync to catch up: 2021-02-11T06:48:37.523 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000002.94.5, , 00000000001.95.5, 00000000001.111.5, 00000000001.71.5, , , 00000000001.69.5, 00000000001.109.5, 00000000002.89.5] 2021-02-11T06:48:37.523 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000002.94.5, , 00000000001.95.5, 00000000001.111.5, 00000000001.71.5, , 00000000001.85.5, 00000000001.69.5, 00000000001.109.5, 00000000002.89.5] 2021-02-11T06:48:42.526 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:48:42.521+0000 7fc3e8bf7080 1 bucket sync caught up with source: 2021-02-11T06:48:42.526 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000002.94.5, , 00000000001.95.5, 00000000001.111.5, 00000000001.71.5, , 00000000001.85.5, 00000000001.69.5, 00000000001.109.5, 00000000002.89.5] 2021-02-11T06:48:42.527 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000002.94.5, , 00000000001.95.5, 00000000001.111.5, 00000000001.71.5, , 00000000001.85.5, 00000000001.69.5, 00000000001.109.5, 00000000002.89.5] 2021-02-11T06:48:42.527 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:48:42.521+0000 7fc3e8bf7080 0 bucket checkpoint complete 2021-02-11T06:48:42.561 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2021-02-11T06:48:31.813849Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'xscaju-72_notif', 'bucket': {'name': 'xscaju-72', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::xscaju-72', 'id': '5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.23'}, 'object': {'key': '0', 'size': 0, 'etag': '2280977e', 'versionId': '', 'sequencer': '3FD324601C4F593B', 'metadata': [], 'tags': []}}, 'eventId': '1613026111.995708.2280977e', 'opaqueData': ''} 2021-02-11T06:48:42.561 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2021-02-11T06:48:31.833419Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'xscaju-72_notif', 'bucket': {'name': 'xscaju-72', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::xscaju-72', 'id': '5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.23'}, 'object': {'key': '1', 'size': 0, 'etag': '91030e8c', 'versionId': '', 'sequencer': '3FD324609E4E5C3B', 'metadata': [], 'tags': []}}, 'eventId': '1613026111.995905.91030e8c', 'opaqueData': ''} 2021-02-11T06:48:42.561 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2021-02-11T06:48:32.053739Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'xscaju-72_notif', 'bucket': {'name': 'xscaju-72', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::xscaju-72', 'id': '5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.23'}, 'object': {'key': '6', 'size': 0, 'etag': '8e0d0ffa', 'versionId': '', 'sequencer': '40D32460896AA708', 'metadata': [], 'tags': []}}, 'eventId': '1613026112.145189.8e0d0ffa', 'opaqueData': ''} 2021-02-11T06:48:42.562 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2021-02-11T06:48:32.009490Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'xscaju-72_notif', 'bucket': {'name': 'xscaju-72', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::xscaju-72', 'id': '5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.23'}, 'object': {'key': '5', 'size': 0, 'etag': '1ffca13e', 'versionId': '', 'sequencer': '40D32460646CAB08', 'metadata': [], 'tags': []}}, 'eventId': '1613026112.145452.1ffca13e', 'opaqueData': ''} 2021-02-11T06:48:42.562 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2021-02-11T06:48:32.186073Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'xscaju-72_notif', 'bucket': {'name': 'xscaju-72', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::xscaju-72', 'id': '5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.23'}, 'object': {'key': '9', 'size': 0, 'etag': '3cba964f', 'versionId': '', 'sequencer': '40D32460738D3A11', 'metadata': [], 'tags': []}}, 'eventId': '1613026112.289049.3cba964f', 'opaqueData': ''} 2021-02-11T06:48:42.562 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2021-02-11T06:48:32.097584Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'xscaju-72_notif', 'bucket': {'name': 'xscaju-72', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::xscaju-72', 'id': '5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.23'}, 'object': {'key': '7', 'size': 0, 'etag': '4dcd3734', 'versionId': '', 'sequencer': '40D32460B8C02917', 'metadata': [], 'tags': []}}, 'eventId': '1613026112.388612.4dcd3734', 'opaqueData': ''} 2021-02-11T06:48:42.562 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2021-02-11T06:48:32.141841Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'xscaju-72_notif', 'bucket': {'name': 'xscaju-72', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::xscaju-72', 'id': '5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.23'}, 'object': {'key': '8', 'size': 0, 'etag': '095c0c67', 'versionId': '', 'sequencer': '40D32460C20E7317', 'metadata': [], 'tags': []}}, 'eventId': '1613026112.393416.095c0c67', 'opaqueData': ''} 2021-02-11T06:48:42.562 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2021-02-11T06:48:31.921594Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'xscaju-72_notif', 'bucket': {'name': 'xscaju-72', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::xscaju-72', 'id': '5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.23'}, 'object': {'key': '3', 'size': 0, 'etag': '58ab693e', 'versionId': '', 'sequencer': '40D3246006FA7817', 'metadata': [], 'tags': []}}, 'eventId': '1613026112.393804.58ab693e', 'opaqueData': ''} 2021-02-11T06:48:42.562 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2021-02-11T06:48:31.965862Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'xscaju-72_notif', 'bucket': {'name': 'xscaju-72', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::xscaju-72', 'id': '5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.23'}, 'object': {'key': '4', 'size': 0, 'etag': '9fa556a7', 'versionId': '', 'sequencer': '40D3246011B5C519', 'metadata': [], 'tags': []}}, 'eventId': '1613026112.432387.9fa556a7', 'opaqueData': ''} 2021-02-11T06:48:42.562 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2021-02-11T06:48:31.878016Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'xscaju-72_notif', 'bucket': {'name': 'xscaju-72', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::xscaju-72', 'id': '5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.23'}, 'object': {'key': '2', 'size': 0, 'etag': '3a99d597', 'versionId': '', 'sequencer': '46D32460E9EBB638', 'metadata': [], 'tags': []}}, 'eventId': '1613026118.951512.3a99d597', 'opaqueData': ''} 2021-02-11T06:48:42.572 WARNING:tasks.rgw_multi.tests_ps:superfluous records are found 2021-02-11T06:48:42.821 INFO:tasks.rgw_multisite_tests:test s3 records fetching ... ok 2021-02-11T06:48:42.821 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:48:42.821 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:48:42.822 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:48:42.864 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:48:42.864 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:48:43.086 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=0d061ff0-ede8-4baa-beef-d975d317d505 2021-02-11T06:48:43.124 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613026122.811792_341.1","last_update":"2021-02-11T06:48:42.811792Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:48:43.125 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T06:48:43.125 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:48:43.125 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:48:43.125 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:48:43.166 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:48:43.167 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:48:43.442 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026122.811792_341.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:48:42.811792Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:48:43.443 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T06:48:43.443 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T06:48:43.443 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:48:43.443 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:48:43.443 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:48:43.482 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:48:43.482 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:48:43.776 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026122.811792_341.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:48:42.811792Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:48:43.776 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T06:48:43.776 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:48:43.776 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:48:43.777 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:48:43.777 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:48:43.817 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:48:43.817 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:48:44.070 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026122.811792_341.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:48:42.811792Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:48:44.071 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:48:44.071 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:48:44.071 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:48:44.072 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:48:44.113 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:48:44.113 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:48:44.366 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=0d061ff0-ede8-4baa-beef-d975d317d505 2021-02-11T06:48:44.409 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613026122.811792_341.1","last_update":"2021-02-11T06:48:42.811792Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:48:44.409 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:48:44.409 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:48:44.409 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:48:44.410 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:48:44.450 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:48:44.450 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:48:44.718 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026122.811792_341.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:48:42.811792Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:48:44.718 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:48:44.779 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:12.331 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:12.332 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:49:12.374 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:49:12.375 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:49:12.626 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=0d061ff0-ede8-4baa-beef-d975d317d505 2021-02-11T06:49:12.665 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613026124.773328_345.1","last_update":"2021-02-11T06:48:44.773328Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:49:12.666 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:49:12.666 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:12.666 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:12.667 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:49:12.708 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:49:12.708 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:49:12.982 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026124.773328_345.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:48:44.773328Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:49:12.982 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:49:13.062 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:13.062 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:13.062 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:49:13.103 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:49:13.104 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:49:13.361 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=0d061ff0-ede8-4baa-beef-d975d317d505 2021-02-11T06:49:13.403 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613026124.773328_345.1","last_update":"2021-02-11T06:48:44.773328Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:49:13.404 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:49:13.404 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:13.404 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:13.404 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:49:13.443 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:49:13.443 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:49:13.716 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026124.773328_345.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:48:44.773328Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:49:13.717 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:49:13.899 INFO:tasks.rgw_multisite_tests:test s3 notification set/get/delete ... ok 2021-02-11T06:49:13.899 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:13.899 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:13.900 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:49:13.937 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:49:13.937 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:49:14.194 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=0d061ff0-ede8-4baa-beef-d975d317d505 2021-02-11T06:49:14.232 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613026153.885312_347.1","last_update":"2021-02-11T06:49:13.885312Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:49:14.233 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T06:49:14.233 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:14.233 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:14.234 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:49:14.279 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:49:14.280 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:49:14.551 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026153.885312_347.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:49:13.885312Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:49:14.553 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T06:49:14.554 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T06:49:14.555 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:14.555 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:14.556 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:49:14.614 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:49:14.614 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:49:14.900 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026153.885312_347.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:49:13.885312Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:49:14.900 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T06:49:14.900 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:49:14.900 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:14.900 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:14.901 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:49:14.946 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:49:14.946 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:49:15.212 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026153.885312_347.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:49:13.885312Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:49:15.212 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:49:15.213 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:15.213 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:15.213 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:49:15.253 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:49:15.254 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:49:15.483 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=0d061ff0-ede8-4baa-beef-d975d317d505 2021-02-11T06:49:15.522 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613026153.885312_347.1","last_update":"2021-02-11T06:49:13.885312Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:49:15.522 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:49:15.522 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:15.522 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:15.522 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:49:15.562 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:49:15.562 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:49:15.820 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026153.885312_347.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:49:13.885312Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:49:15.820 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:49:16.049 INFO:tasks.rgw_multisite_tests:test s3 topics set/get/delete on master ... ok 2021-02-11T06:49:16.049 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:16.049 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:16.049 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:49:16.089 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:49:16.090 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:49:16.331 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=0d061ff0-ede8-4baa-beef-d975d317d505 2021-02-11T06:49:16.371 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613026153.885312_347.1","last_update":"2021-02-11T06:49:13.885312Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:49:16.372 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T06:49:16.372 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:16.372 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:16.372 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:49:16.413 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:49:16.413 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:49:16.685 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026153.885312_347.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:49:13.885312Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:49:16.686 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T06:49:16.686 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T06:49:16.686 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:16.686 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:16.686 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:49:16.726 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:49:16.726 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:49:17.007 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026153.885312_347.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:49:13.885312Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:49:17.008 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T06:49:17.008 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:49:17.008 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:17.008 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:17.008 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:49:17.050 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:49:17.050 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:49:17.319 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026153.885312_347.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:49:13.885312Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:49:17.319 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:49:17.319 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:17.319 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:17.320 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:49:17.359 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:49:17.359 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:49:17.613 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=0d061ff0-ede8-4baa-beef-d975d317d505 2021-02-11T06:49:17.648 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613026153.885312_347.1","last_update":"2021-02-11T06:49:13.885312Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:49:17.648 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:49:17.648 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:17.648 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:17.648 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:49:17.689 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:49:17.689 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:49:17.957 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026153.885312_347.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:49:13.885312Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:49:17.957 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:49:17.957 INFO:tasks.rgw_multisite_tests:test s3 topics with secret set/get/delete on master ... ok 2021-02-11T06:49:17.958 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:17.958 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:17.958 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:49:17.999 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:49:17.999 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:49:18.233 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=0d061ff0-ede8-4baa-beef-d975d317d505 2021-02-11T06:49:18.272 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613026153.885312_347.1","last_update":"2021-02-11T06:49:13.885312Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:49:18.272 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T06:49:18.272 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:18.272 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:18.273 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:49:18.317 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:49:18.317 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:49:18.572 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026153.885312_347.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:49:13.885312Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:49:18.572 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T06:49:18.572 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T06:49:18.573 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:18.573 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:18.573 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:49:18.615 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:49:18.615 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:49:18.899 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026153.885312_347.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:49:13.885312Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:49:18.900 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T06:49:18.900 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:49:18.900 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:18.900 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:18.900 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:49:18.941 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:49:18.941 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:49:19.206 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026153.885312_347.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:49:13.885312Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:49:19.207 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:49:19.207 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:19.207 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:19.207 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:49:19.249 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:49:19.249 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:49:19.505 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=0d061ff0-ede8-4baa-beef-d975d317d505 2021-02-11T06:49:19.546 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613026153.885312_347.1","last_update":"2021-02-11T06:49:13.885312Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:49:19.546 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:49:19.546 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:19.547 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:19.547 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:49:19.588 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:49:19.588 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:49:19.844 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026153.885312_347.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:49:13.885312Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:49:19.845 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:49:20.287 INFO:tasks.rgw_multisite_tests:test s3 notification set/get/delete on master ... ok 2021-02-11T06:49:22.299 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests_ps.test_ps_s3_notification_filter_on_master ... ok 2021-02-11T06:49:22.300 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests_ps.test_ps_s3_notification_filter ... ok 2021-02-11T06:49:22.300 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:22.300 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:22.300 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:49:22.343 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:49:22.343 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:49:22.601 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=0d061ff0-ede8-4baa-beef-d975d317d505 2021-02-11T06:49:22.894 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613026160.231529_353.1","last_update":"2021-02-11T06:49:20.231529Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:49:22.894 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T06:49:22.895 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:22.895 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:22.895 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:49:22.934 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:49:22.934 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:49:23.223 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026160.231529_353.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:49:20.231529Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:49:23.223 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T06:49:23.223 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T06:49:23.223 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:23.224 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:23.224 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:49:23.264 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:49:23.264 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:49:23.541 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026160.231529_353.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:49:20.231529Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:49:23.542 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T06:49:23.542 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:49:23.542 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:23.542 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:23.542 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:49:23.583 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:49:23.583 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:49:23.868 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026160.231529_353.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:49:20.231529Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:49:23.868 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:49:23.868 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:23.869 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:23.869 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:49:23.909 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:49:23.909 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:49:24.155 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=0d061ff0-ede8-4baa-beef-d975d317d505 2021-02-11T06:49:24.190 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613026160.231529_353.1","last_update":"2021-02-11T06:49:20.231529Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:49:24.191 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:49:24.191 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:24.191 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:24.191 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:49:24.232 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:49:24.232 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:49:24.470 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026160.231529_353.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:49:20.231529Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:49:24.471 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:49:24.825 INFO:tasks.rgw_multisite_tests:test s3 notification set/get/delete on master ... ok 2021-02-11T06:49:24.825 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests_ps.test_objcet_timing ... ok 2021-02-11T06:49:24.826 INFO:tasks.rgw_multisite_tests:test pushing amqp s3 notification on master ... ok 2021-02-11T06:49:24.826 INFO:tasks.rgw_multisite_tests:test reservation cleanup after gateway crash ... ok 2021-02-11T06:49:24.826 INFO:tasks.rgw_multisite_tests:test gateway recovery of persistent notifications ... ok 2021-02-11T06:49:24.826 INFO:tasks.rgw_multisite_tests:test pushing persistent notification via two gateways ... ok 2021-02-11T06:49:24.826 INFO:tasks.rgw_multisite_tests:test pushing persistent notification when one of the endpoints has error ... ok 2021-02-11T06:49:24.827 INFO:tasks.rgw_multisite_tests:test pushing persistent notification http ... ok 2021-02-11T06:49:24.827 INFO:tasks.rgw_multisite_tests:test pushing persistent notification amqp ... ok 2021-02-11T06:49:24.827 INFO:tasks.rgw_multisite_tests:test pushing persistent notification of large notifications ... ok 2021-02-11T06:49:24.827 INFO:tasks.rgw_multisite_tests:test pushing persistent notification pushback ... ok 2021-02-11T06:49:24.828 INFO:tasks.rgw_multisite_tests:test pushing kafka s3 notification on master ... ok 2021-02-11T06:49:24.828 INFO:tasks.rgw_multisite_tests:test pushing kafka s3 notification on master ... ok 2021-02-11T06:49:24.828 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests_ps.test_ps_s3_notification_push_kafka_security_ssl ... ok 2021-02-11T06:49:24.829 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests_ps.test_ps_s3_notification_push_kafka_security_ssl_sasl ... ok 2021-02-11T06:49:24.829 INFO:tasks.rgw_multisite_tests:test deletion of multiple keys on master ... ok 2021-02-11T06:49:24.829 INFO:tasks.rgw_multisite_tests:test pushing http s3 notification on master ... ok 2021-02-11T06:49:24.829 INFO:tasks.rgw_multisite_tests:test that opaque id set in topic, is sent in notification ... ok 2021-02-11T06:49:24.829 INFO:tasks.rgw_multisite_tests:test that opaque id set in topic, is sent in notification on master ... ok 2021-02-11T06:49:24.830 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:24.830 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:24.830 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:49:24.871 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:49:24.871 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:49:25.114 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=0d061ff0-ede8-4baa-beef-d975d317d505 2021-02-11T06:49:25.155 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613026164.813304_359.1","last_update":"2021-02-11T06:49:24.813304Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:49:25.155 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T06:49:25.156 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:25.156 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:25.156 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:49:25.196 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:49:25.197 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:49:25.480 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026164.813304_359.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:49:24.813304Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:49:25.480 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T06:49:25.481 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T06:49:25.481 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:25.481 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:25.481 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:49:25.522 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:49:25.522 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:49:25.817 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026164.813304_359.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:49:24.813304Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:49:25.817 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T06:49:25.817 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:49:25.818 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:25.818 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:25.818 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:49:25.858 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:49:25.858 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:49:26.126 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026164.813304_359.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:49:24.813304Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:49:26.126 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:49:26.127 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:26.127 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:26.127 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:49:26.166 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:49:26.166 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:49:26.411 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=0d061ff0-ede8-4baa-beef-d975d317d505 2021-02-11T06:49:26.450 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613026164.813304_359.1","last_update":"2021-02-11T06:49:24.813304Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:49:26.451 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:49:26.451 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:26.451 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:26.451 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:49:26.490 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:49:26.490 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:49:26.737 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026164.813304_359.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:49:24.813304Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:49:26.738 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:49:26.758 INFO:tasks.rgw_multisite_tests:test set/get/delete of topic ... ok 2021-02-11T06:49:26.758 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:26.758 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:26.758 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:49:26.799 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:49:26.799 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:49:27.037 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=0d061ff0-ede8-4baa-beef-d975d317d505 2021-02-11T06:49:27.078 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613026164.813304_359.1","last_update":"2021-02-11T06:49:24.813304Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:49:27.079 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T06:49:27.079 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:27.079 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:27.079 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:49:27.122 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:49:27.122 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:49:27.378 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026164.813304_359.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:49:24.813304Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:49:27.378 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T06:49:27.378 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T06:49:27.378 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:27.379 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:27.379 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:49:27.417 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:49:27.417 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:49:27.711 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026164.813304_359.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:49:24.813304Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:49:27.711 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T06:49:27.711 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:49:27.711 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:27.711 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:27.712 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:49:27.752 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:49:27.752 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:49:28.005 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026164.813304_359.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:49:24.813304Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:49:28.005 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:49:28.005 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:28.005 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:28.006 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:49:28.047 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:49:28.047 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:49:28.294 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=0d061ff0-ede8-4baa-beef-d975d317d505 2021-02-11T06:49:28.333 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613026164.813304_359.1","last_update":"2021-02-11T06:49:24.813304Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:49:28.334 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:49:28.334 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:28.334 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:28.334 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:49:28.375 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:49:28.375 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:49:28.646 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026164.813304_359.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:49:24.813304Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:49:28.646 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:49:28.666 INFO:tasks.rgw_multisite_tests:test set topic with endpoint ... ok 2021-02-11T06:49:28.667 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:28.667 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:28.667 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:49:28.708 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:49:28.708 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:49:28.970 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=0d061ff0-ede8-4baa-beef-d975d317d505 2021-02-11T06:49:29.011 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613026164.813304_359.1","last_update":"2021-02-11T06:49:24.813304Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:49:29.011 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T06:49:29.011 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:29.012 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:29.012 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:49:29.053 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:49:29.053 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:49:29.334 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026164.813304_359.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:49:24.813304Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:49:29.334 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T06:49:29.334 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T06:49:29.335 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:29.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:29.335 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:49:29.377 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:49:29.377 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:49:29.653 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026164.813304_359.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:49:24.813304Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:49:29.653 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T06:49:29.653 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:49:29.653 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:29.654 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:29.654 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:49:29.694 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:49:29.694 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:49:29.954 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026164.813304_359.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:49:24.813304Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:49:30.148 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:49:30.148 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:30.149 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:30.149 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:49:30.192 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:49:30.192 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:49:30.452 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=0d061ff0-ede8-4baa-beef-d975d317d505 2021-02-11T06:49:30.489 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613026164.813304_359.1","last_update":"2021-02-11T06:49:24.813304Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:49:30.489 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:49:30.490 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:30.490 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:30.490 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:49:30.531 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:49:30.531 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:49:30.805 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026164.813304_359.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:49:24.813304Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:49:30.806 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:49:30.869 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:30.869 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:30.869 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:49:30.909 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:49:30.909 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:49:31.143 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=0d061ff0-ede8-4baa-beef-d975d317d505 2021-02-11T06:49:31.184 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613026170.861151_363.1","last_update":"2021-02-11T06:49:30.861151Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:49:31.185 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:49:31.185 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:31.185 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:31.185 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:49:31.225 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:49:31.226 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:49:31.486 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026170.861151_363.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:49:30.861151Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:49:31.487 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:49:31.632 INFO:tasks.rgw_multisite_tests:test set/get/delete of notification ... ok 2021-02-11T06:49:31.632 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:31.633 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:31.633 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:49:31.670 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:49:31.670 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:49:31.902 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=0d061ff0-ede8-4baa-beef-d975d317d505 2021-02-11T06:49:31.941 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613026171.619391_365.1","last_update":"2021-02-11T06:49:31.619391Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:49:31.941 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T06:49:31.941 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:31.941 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:31.941 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:49:31.981 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:49:31.982 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:49:32.255 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026171.619391_365.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:49:31.619391Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:49:33.531 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T06:49:33.531 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T06:49:33.531 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:33.531 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:33.531 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:49:33.574 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:49:33.574 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:49:33.865 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026171.619391_365.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:49:31.619391Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:49:33.866 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T06:49:33.866 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:49:33.866 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:33.866 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:33.866 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:49:33.906 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:49:33.906 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:49:34.179 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026171.619391_365.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:49:31.619391Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:49:34.179 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:49:34.179 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:34.180 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:34.180 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:49:34.221 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:49:34.221 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:49:34.461 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=0d061ff0-ede8-4baa-beef-d975d317d505 2021-02-11T06:49:34.497 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613026171.619391_365.1","last_update":"2021-02-11T06:49:31.619391Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:49:34.497 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:49:34.498 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:34.498 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:34.498 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:49:34.539 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:49:34.539 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:49:34.796 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026171.619391_365.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:49:31.619391Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:49:34.797 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:49:34.863 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:34.863 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:34.864 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:49:34.906 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:49:34.906 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:49:35.148 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=0d061ff0-ede8-4baa-beef-d975d317d505 2021-02-11T06:49:35.188 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613026174.858657_369.1","last_update":"2021-02-11T06:49:34.858657Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:49:35.188 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:49:35.189 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:35.189 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:35.189 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:49:35.231 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:49:35.232 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:49:35.498 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026174.858657_369.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:49:34.858657Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:49:35.498 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:49:35.646 INFO:tasks.rgw_multisite_tests:test set/get/delete of notification on specific events ... ok 2021-02-11T06:49:35.646 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:35.646 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:35.646 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:49:35.685 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:49:35.685 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:49:35.935 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=0d061ff0-ede8-4baa-beef-d975d317d505 2021-02-11T06:49:35.975 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613026175.634013_371.1","last_update":"2021-02-11T06:49:35.634013Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:49:35.975 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T06:49:35.975 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:35.975 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:35.975 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:49:36.016 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:49:36.017 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:49:36.290 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026175.634013_371.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:49:35.634013Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:49:36.290 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T06:49:36.290 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T06:49:36.291 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:36.291 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:36.291 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:49:36.331 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:49:36.331 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:49:36.623 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026175.634013_371.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:49:35.634013Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:49:36.624 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T06:49:36.624 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:49:36.624 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:36.624 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:36.624 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:49:36.665 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:49:36.665 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:49:36.926 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026175.634013_371.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:49:35.634013Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:49:36.926 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:49:36.927 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:36.927 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:36.927 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:49:36.968 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:49:36.968 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:49:37.239 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=0d061ff0-ede8-4baa-beef-d975d317d505 2021-02-11T06:49:37.276 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613026175.634013_371.1","last_update":"2021-02-11T06:49:35.634013Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:49:37.276 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:49:37.276 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:37.276 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:37.277 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:49:37.317 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:49:37.317 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:49:37.595 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026175.634013_371.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:49:35.634013Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:49:37.596 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:49:37.656 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:37.656 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:37.656 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:49:37.696 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:49:37.696 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:49:37.934 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=0d061ff0-ede8-4baa-beef-d975d317d505 2021-02-11T06:49:37.974 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613026177.652183_375.1","last_update":"2021-02-11T06:49:37.652183Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:49:37.974 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:49:37.974 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:37.975 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:37.975 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:49:38.014 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:49:38.014 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:49:38.281 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026177.652183_375.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:49:37.652183Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:49:38.282 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:49:38.722 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-81', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:49:38.723 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-81', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:49:38.723 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-81 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:49:38.766 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:49:38.767 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:49:38.989 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:49:38.984+0000 7fcb127b5080 1 bucket sync caught up with source: 2021-02-11T06:49:38.989 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000002.92.5, , 00000000001.98.5, 00000000001.99.5, 00000000001.99.5, , 00000000001.101.5, 00000000001.100.5, 00000000001.209.5, 00000000002.115.5] 2021-02-11T06:49:38.989 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000002.92.5, , 00000000001.98.5, 00000000001.99.5, 00000000001.99.5, , 00000000001.101.5, 00000000001.100.5, 00000000001.209.5, 00000000002.115.5] 2021-02-11T06:49:38.989 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:49:38.984+0000 7fcb127b5080 0 bucket checkpoint complete 2021-02-11T06:49:39.030 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "4" type: "OBJECT_CREATE" 2021-02-11T06:49:39.030 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "2" type: "OBJECT_CREATE" 2021-02-11T06:49:39.031 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "1" type: "OBJECT_CREATE" 2021-02-11T06:49:39.031 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "3" type: "OBJECT_CREATE" 2021-02-11T06:49:39.031 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "0" type: "OBJECT_CREATE" 2021-02-11T06:49:39.031 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "6" type: "OBJECT_CREATE" 2021-02-11T06:49:39.031 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "5" type: "OBJECT_CREATE" 2021-02-11T06:49:39.031 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "9" type: "OBJECT_CREATE" 2021-02-11T06:49:39.031 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "7" type: "OBJECT_CREATE" 2021-02-11T06:49:39.031 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "8" type: "OBJECT_CREATE" 2021-02-11T06:49:39.037 DEBUG:tasks.rgw_multi.tests_ps:superfluous events are found 2021-02-11T06:49:39.162 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-81', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:49:39.162 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-81', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:49:39.162 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-81 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:49:39.215 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:49:39.215 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:49:39.432 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:49:39.424+0000 7fc157b64080 1 bucket sync caught up with source: 2021-02-11T06:49:39.432 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000004.96.5, , 00000000002.100.5, 00000000002.101.5, 00000000002.101.5, , 00000000002.103.5, 00000000002.102.5, 00000000002.211.5, 00000000004.119.5] 2021-02-11T06:49:39.432 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000004.96.5, , 00000000002.100.5, 00000000002.101.5, 00000000002.101.5, , 00000000002.103.5, 00000000002.102.5, 00000000002.211.5, 00000000004.119.5] 2021-02-11T06:49:39.432 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:49:39.424+0000 7fc157b64080 0 bucket checkpoint complete 2021-02-11T06:49:39.603 INFO:tasks.rgw_multisite_tests:test set/get/delete of subscription ... ok 2021-02-11T06:49:39.604 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:39.604 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:39.604 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:49:39.643 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:49:39.644 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:49:39.888 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=0d061ff0-ede8-4baa-beef-d975d317d505 2021-02-11T06:49:39.926 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613026179.591735_377.1","last_update":"2021-02-11T06:49:39.591735Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:49:39.927 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T06:49:39.927 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:39.927 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:49:39.927 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:49:39.968 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:49:39.968 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:49:40.243 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026179.591735_377.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:49:39.591735Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:50:06.685 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T06:50:06.685 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T06:50:06.685 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:50:06.685 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:50:06.686 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:50:06.727 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:50:06.727 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:50:07.012 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026179.591735_377.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:49:39.591735Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:50:07.012 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T06:50:07.013 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:50:07.013 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:50:07.013 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:50:07.013 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:50:07.053 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:50:07.053 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:50:07.314 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026179.591735_377.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:49:39.591735Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:50:07.315 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:50:07.315 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:50:07.315 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:50:07.315 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:50:07.354 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:50:07.355 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:50:07.615 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=0d061ff0-ede8-4baa-beef-d975d317d505 2021-02-11T06:50:07.656 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613026179.591735_377.1","last_update":"2021-02-11T06:49:39.591735Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:50:07.656 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:50:07.656 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:50:07.656 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:50:07.657 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:50:07.697 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:50:07.698 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:50:07.970 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026179.591735_377.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:49:39.591735Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:50:07.970 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:50:07.982 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['topic', 'list', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:50:07.982 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'topic', 'list', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:50:07.982 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 topic list --uid rgw-multisite-test-user --rgw-zone test-zone4 --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T06:50:08.048 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:50:08.049 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:50:08.320 INFO:teuthology.orchestra.run.smithi172.stdout:{"topics":[{"topic":{"user":"","name":"xscaju-82_topic","dest":{"bucket_name":"","oid_prefix":"","push_endpoint":"","push_endpoint_args":"","push_endpoint_topic":"xscaju-82_topic","stored_secret":"false","persistent":"false"},"arn":"arn:aws:sns:test-zonegroup::xscaju-82_topic","opaqueData":""},"subs":[]}]} 2021-02-11T06:50:08.320 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['topic', 'get', '--uid', 'rgw-multisite-test-user', '--topic', 'xscaju-82_topic', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:50:08.320 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'topic', 'get', '--uid', 'rgw-multisite-test-user', '--topic', 'xscaju-82_topic', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:50:08.321 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 topic get --uid rgw-multisite-test-user --topic xscaju-82_topic --rgw-zone test-zone4 --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T06:50:08.403 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:50:08.403 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:50:08.667 INFO:teuthology.orchestra.run.smithi172.stdout:{"topic":{"user":"","name":"xscaju-82_topic","dest":{"bucket_name":"","oid_prefix":"","push_endpoint":"","push_endpoint_args":"","push_endpoint_topic":"xscaju-82_topic","stored_secret":"false","persistent":"false"},"arn":"arn:aws:sns:test-zonegroup::xscaju-82_topic","opaqueData":""},"subs":[]} 2021-02-11T06:50:08.688 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['topic', 'list', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:50:08.688 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'topic', 'list', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:50:08.688 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 topic list --uid rgw-multisite-test-user --rgw-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T06:50:08.726 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:50:08.726 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:50:09.035 INFO:teuthology.orchestra.run.smithi061.stdout:{"topics":[{"topic":{"user":"","name":"xscaju-82_topic","dest":{"bucket_name":"","oid_prefix":"","push_endpoint":"amqp://127.0.0.1:7001/vhost_1","push_endpoint_args":"Attributes.entry.1.key=push-endpoint&Attributes.entry.1.value=amqp://127.0.0.1:7001/vhost_1&Attributes.entry.2.key=amqp-exchange&Attributes.entry.2.value=amqp.direct&Attributes.entry.3.key=amqp-ack-level&Attributes.entry.3.value=none&Version=2010-03-31&amqp-ack-level=none&amqp-exchange=amqp.direct&push-endpoint=amqp://127.0.0.1:7001/vhost_1","push_endpoint_topic":"xscaju-82_topic","stored_secret":"false","persistent":"false"},"arn":"arn:aws:sns:test-zonegroup::xscaju-82_topic","opaqueData":""},"subs":[]}]} 2021-02-11T06:50:09.035 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['topic', 'get', '--uid', 'rgw-multisite-test-user', '--topic', 'xscaju-82_topic', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:50:09.036 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'topic', 'get', '--uid', 'rgw-multisite-test-user', '--topic', 'xscaju-82_topic', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:50:09.036 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 topic get --uid rgw-multisite-test-user --topic xscaju-82_topic --rgw-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T06:50:09.116 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:50:09.117 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:50:09.408 INFO:teuthology.orchestra.run.smithi061.stdout:{"topic":{"user":"","name":"xscaju-82_topic","dest":{"bucket_name":"","oid_prefix":"","push_endpoint":"amqp://127.0.0.1:7001/vhost_1","push_endpoint_args":"Attributes.entry.1.key=push-endpoint&Attributes.entry.1.value=amqp://127.0.0.1:7001/vhost_1&Attributes.entry.2.key=amqp-exchange&Attributes.entry.2.value=amqp.direct&Attributes.entry.3.key=amqp-ack-level&Attributes.entry.3.value=none&Version=2010-03-31&amqp-ack-level=none&amqp-exchange=amqp.direct&push-endpoint=amqp://127.0.0.1:7001/vhost_1","push_endpoint_topic":"xscaju-82_topic","stored_secret":"false","persistent":"false"},"arn":"arn:aws:sns:test-zonegroup::xscaju-82_topic","opaqueData":""},"subs":[]} 2021-02-11T06:50:09.455 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:50:09.456 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:50:09.456 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:50:09.490 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:50:09.490 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:50:09.730 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=0d061ff0-ede8-4baa-beef-d975d317d505 2021-02-11T06:50:09.768 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613026209.451303_381.1","last_update":"2021-02-11T06:50:09.451303Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:50:09.768 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:50:09.769 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:50:09.769 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:50:09.769 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:50:09.810 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:50:09.810 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:50:10.057 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026209.451303_381.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:50:09.451303Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:50:10.058 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:50:10.081 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['subscription', 'get', '--uid', 'rgw-multisite-test-user', '--subscription', 'xscaju-82_sub', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:50:10.081 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'subscription', 'get', '--uid', 'rgw-multisite-test-user', '--subscription', 'xscaju-82_sub', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:50:10.081 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 subscription get --uid rgw-multisite-test-user --subscription xscaju-82_sub --rgw-zone test-zone4 --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T06:50:10.146 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:50:10.146 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:50:10.398 INFO:teuthology.orchestra.run.smithi172.stdout:{"user":"","name":"xscaju-82_sub","topic":"xscaju-82_topic","dest":{"bucket_name":"pubsub-rgw-multisite-test-user-xscaju-82_topic","oid_prefix":"xscaju-82_sub/","push_endpoint":"","push_endpoint_args":"topic=xscaju-82_topic","push_endpoint_topic":"xscaju-82_topic","stored_secret":"false","persistent":"false"},"s3_id":""} 2021-02-11T06:50:15.359 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-82', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:50:15.359 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-82', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:50:15.359 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-82 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:50:15.404 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:50:15.404 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:50:15.640 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:50:15.639+0000 7faac746c080 1 bucket sync caught up with source: 2021-02-11T06:50:15.641 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [00000000011.126.5, 00000000011.93.5, 00000000010.127.5, 00000000010.95.5, 00000000009.85.5, 00000000007.68.5, 00000000010.112.5, 00000000010.118.5, 00000000011.137.5, 00000000011.124.5, 00000000011.128.5] 2021-02-11T06:50:15.641 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [00000000011.126.5, 00000000011.93.5, 00000000010.127.5, 00000000010.95.5, 00000000009.85.5, 00000000007.68.5, 00000000010.112.5, 00000000010.118.5, 00000000011.137.5, 00000000011.124.5, 00000000011.128.5] 2021-02-11T06:50:15.641 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:50:15.639+0000 7faac746c080 0 bucket checkpoint complete 2021-02-11T06:50:15.669 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['subscription', 'pull', '--uid', 'rgw-multisite-test-user', '--subscription', 'xscaju-82_sub', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:50:15.670 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'subscription', 'pull', '--uid', 'rgw-multisite-test-user', '--subscription', 'xscaju-82_sub', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:50:15.670 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 subscription pull --uid rgw-multisite-test-user --subscription xscaju-82_sub --rgw-zone test-zone4 --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T06:50:15.752 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:50:15.753 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:50:16.050 INFO:teuthology.orchestra.run.smithi172.stdout:{"next_marker":"xscaju-82_sub/1613026214.941313.203b4715","is_truncated":"true","events":[{"id":"1613026210.601976.d7bbccd5","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:10.601976Z","info":{"attrs":{"mtime":"2021-02-11T06:50:10.466234Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"2"}}},{"id":"1613026210.603160.788cbb52","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:10.603160Z","info":{"attrs":{"mtime":"2021-02-11T06:50:10.423510Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"1"}}},{"id":"1613026210.684252.7121c066","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:10.684252Z","info":{"attrs":{"mtime":"2021-02-11T06:50:10.403393Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"0"}}},{"id":"1613026210.684427.c61a196e","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:10.684427Z","info":{"attrs":{"mtime":"2021-02-11T06:50:10.510181Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"3"}}},{"id":"1613026210.793246.2dcdf9ba","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:10.793246Z","info":{"attrs":{"mtime":"2021-02-11T06:50:10.598206Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"5"}}},{"id":"1613026210.830470.5b62ac69","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:10.830470Z","info":{"attrs":{"mtime":"2021-02-11T06:50:10.646304Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"6"}}},{"id":"1613026210.840474.1f11c500","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:10.840474Z","info":{"attrs":{"mtime":"2021-02-11T06:50:10.691104Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"7"}}},{"id":"1613026210.886696.d348f6b6","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:10.886696Z","info":{"attrs":{"mtime":"2021-02-11T06:50:10.782712Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"9"}}},{"id":"1613026210.943278.9f9c5975","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:10.943278Z","info":{"attrs":{"mtime":"2021-02-11T06:50:10.877886Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"11"}}},{"id":"1613026210.944074.d348f6b6","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:10.944074Z","info":{"attrs":{"mtime":"2021-02-11T06:50:10.782712Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"9"}}},{"id":"1613026210.990517.b88a1e7d","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:10.990517Z","info":{"attrs":{"mtime":"2021-02-11T06:50:10.553500Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"4"}}},{"id":"1613026210.994110.be776e83","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:10.994110Z","info":{"attrs":{"mtime":"2021-02-11T06:50:10.735860Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"8"}}},{"id":"1613026211.150429.9164d9b6","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:11.150429Z","info":{"attrs":{"mtime":"2021-02-11T06:50:11.057710Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"15"}}},{"id":"1613026211.151613.84473bc3","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:11.151613Z","info":{"attrs":{"mtime":"2021-02-11T06:50:11.013604Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"14"}}},{"id":"1613026211.194928.e7dd6115","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:11.194928Z","info":{"attrs":{"mtime":"2021-02-11T06:50:10.969943Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"13"}}},{"id":"1613026211.228130.d57982e2","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:11.228130Z","info":{"attrs":{"mtime":"2021-02-11T06:50:11.105073Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"16"}}},{"id":"1613026211.228358.147fe6b4","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:11.228358Z","info":{"attrs":{"mtime":"2021-02-11T06:50:10.927011Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"12"}}},{"id":"1613026211.268258.7167ef9f","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:11.268258Z","info":{"attrs":{"mtime":"2021-02-11T06:50:10.831780Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"10"}}},{"id":"1613026211.397056.2fe99e2b","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:11.397056Z","info":{"attrs":{"mtime":"2021-02-11T06:50:11.197375Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"18"}}},{"id":"1613026211.397170.d5505fc1","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:11.397170Z","info":{"attrs":{"mtime":"2021-02-11T06:50:11.289579Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"20"}}},{"id":"1613026211.404251.15584d06","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:11.404251Z","info":{"attrs":{"mtime":"2021-02-11T06:50:11.153295Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"17"}}},{"id":"1613026211.442862.ff31d0cd","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:11.442862Z","info":{"attrs":{"mtime":"2021-02-11T06:50:11.334320Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"21"}}},{"id":"1613026211.552179.09bc16eb","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:11.552179Z","info":{"attrs":{"mtime":"2021-02-11T06:50:11.421378Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"23"}}},{"id":"1613026211.552376.91886d6b","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:11.552376Z","info":{"attrs":{"mtime":"2021-02-11T06:50:11.241522Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"19"}}},{"id":"1613026211.552934.9e57e5fa","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:11.552934Z","info":{"attrs":{"mtime":"2021-02-11T06:50:11.377320Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"22"}}},{"id":"1613026211.636320.390b4cbf","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:11.636320Z","info":{"attrs":{"mtime":"2021-02-11T06:50:11.513577Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"25"}}},{"id":"1613026211.648869.611b8483","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:11.648869Z","info":{"attrs":{"mtime":"2021-02-11T06:50:11.465476Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"24"}}},{"id":"1613026211.715035.903ce197","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:11.715035Z","info":{"attrs":{"mtime":"2021-02-11T06:50:11.655722Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"28"}}},{"id":"1613026211.785102.f4d9530c","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:11.785102Z","info":{"attrs":{"mtime":"2021-02-11T06:50:11.610539Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"27"}}},{"id":"1613026211.794303.9c047147","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:11.794303Z","info":{"attrs":{"mtime":"2021-02-11T06:50:11.697796Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"29"}}},{"id":"1613026211.795022.2c2d7073","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:11.795022Z","info":{"attrs":{"mtime":"2021-02-11T06:50:11.557744Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"26"}}},{"id":"1613026211.946078.7efc2ec7","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:11.946078Z","info":{"attrs":{"mtime":"2021-02-11T06:50:11.833369Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"32"}}},{"id":"1613026211.990909.e4dea283","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:11.990909Z","info":{"attrs":{"mtime":"2021-02-11T06:50:11.877809Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"33"}}},{"id":"1613026211.991041.3823fcde","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:11.991041Z","info":{"attrs":{"mtime":"2021-02-11T06:50:11.790258Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"31"}}},{"id":"1613026212.028068.be9059d5","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:12.028068Z","info":{"attrs":{"mtime":"2021-02-11T06:50:11.741699Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"30"}}},{"id":"1613026212.076478.3ef398da","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:12.076478Z","info":{"attrs":{"mtime":"2021-02-11T06:50:11.965691Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"35"}}},{"id":"1613026212.153837.a5c31b7d","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:12.153837Z","info":{"attrs":{"mtime":"2021-02-11T06:50:12.009391Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"36"}}},{"id":"1613026212.203083.4bd04338","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:12.203083Z","info":{"attrs":{"mtime":"2021-02-11T06:50:12.057694Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"37"}}},{"id":"1613026212.209315.6137a83a","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:12.209315Z","info":{"attrs":{"mtime":"2021-02-11T06:50:12.101702Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"38"}}},{"id":"1613026212.240229.073ed26f","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:12.240229Z","info":{"attrs":{"mtime":"2021-02-11T06:50:11.922304Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"34"}}},{"id":"1613026212.382686.f3718497","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:12.382686Z","info":{"attrs":{"mtime":"2021-02-11T06:50:12.233280Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"41"}}},{"id":"1613026212.460126.14355e00","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:12.460126Z","info":{"attrs":{"mtime":"2021-02-11T06:50:12.189451Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"40"}}},{"id":"1613026212.542386.2377d740","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:12.542386Z","info":{"attrs":{"mtime":"2021-02-11T06:50:12.365624Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"44"}}},{"id":"1613026212.592284.713fb4a6","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:12.592284Z","info":{"attrs":{"mtime":"2021-02-11T06:50:12.409855Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"45"}}},{"id":"1613026212.596200.4b15a9b6","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:12.596200Z","info":{"attrs":{"mtime":"2021-02-11T06:50:12.497613Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"47"}}},{"id":"1613026212.636267.c8203ac8","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:12.636267Z","info":{"attrs":{"mtime":"2021-02-11T06:50:12.321738Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"43"}}},{"id":"1613026212.743374.c4c12317","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:12.743374Z","info":{"attrs":{"mtime":"2021-02-11T06:50:12.146208Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"39"}}},{"id":"1613026212.744150.d18b53eb","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:12.744150Z","info":{"attrs":{"mtime":"2021-02-11T06:50:12.673998Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"51"}}},{"id":"1613026212.744371.c58ace16","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:12.744371Z","info":{"attrs":{"mtime":"2021-02-11T06:50:12.277624Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"42"}}},{"id":"1613026212.744578.fc878b57","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:12.744578Z","info":{"attrs":{"mtime":"2021-02-11T06:50:12.629287Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"50"}}},{"id":"1613026212.789891.b5600e34","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:12.789891Z","info":{"attrs":{"mtime":"2021-02-11T06:50:12.587162Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"49"}}},{"id":"1613026212.790941.d1ab7252","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:12.790941Z","info":{"attrs":{"mtime":"2021-02-11T06:50:12.717511Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"52"}}},{"id":"1613026212.793420.fb96724d","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:12.793420Z","info":{"attrs":{"mtime":"2021-02-11T06:50:12.453460Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"46"}}},{"id":"1613026212.796041.b3073fee","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:12.796041Z","info":{"attrs":{"mtime":"2021-02-11T06:50:12.542919Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"48"}}},{"id":"1613026212.940692.83222326","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:12.940692Z","info":{"attrs":{"mtime":"2021-02-11T06:50:12.761531Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"53"}}},{"id":"1613026212.947161.efd6bea4","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:12.947161Z","info":{"attrs":{"mtime":"2021-02-11T06:50:12.849567Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"55"}}},{"id":"1613026212.988409.4bb002f2","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:12.988409Z","info":{"attrs":{"mtime":"2021-02-11T06:50:12.805687Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"54"}}},{"id":"1613026212.994861.b4cc4818","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:12.994861Z","info":{"attrs":{"mtime":"2021-02-11T06:50:12.897836Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"56"}}},{"id":"1613026213.140890.63ff6e11","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:13.140890Z","info":{"attrs":{"mtime":"2021-02-11T06:50:12.941668Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"57"}}},{"id":"1613026213.143621.67116659","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:13.143621Z","info":{"attrs":{"mtime":"2021-02-11T06:50:13.029521Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"59"}}},{"id":"1613026213.143956.bfee43ca","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:13.143956Z","info":{"attrs":{"mtime":"2021-02-11T06:50:13.077705Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"60"}}},{"id":"1613026213.192474.e992b040","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:13.192474Z","info":{"attrs":{"mtime":"2021-02-11T06:50:12.985571Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"58"}}},{"id":"1613026213.350649.963ee7c1","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:13.350649Z","info":{"attrs":{"mtime":"2021-02-11T06:50:13.224507Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"63"}}},{"id":"1613026213.392712.7928bbaa","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:13.392712Z","info":{"attrs":{"mtime":"2021-02-11T06:50:13.122487Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"61"}}},{"id":"1613026213.392941.b79a5625","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:13.392941Z","info":{"attrs":{"mtime":"2021-02-11T06:50:13.323948Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"65"}}},{"id":"1613026213.394901.58e37f5b","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:13.394901Z","info":{"attrs":{"mtime":"2021-02-11T06:50:13.272914Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"64"}}},{"id":"1613026213.428088.69bc003e","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:13.428088Z","info":{"attrs":{"mtime":"2021-02-11T06:50:13.175919Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"62"}}},{"id":"1613026213.476308.afce25a0","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:13.476308Z","info":{"attrs":{"mtime":"2021-02-11T06:50:13.375861Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"66"}}},{"id":"1613026213.545546.29b7a6de","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:13.545546Z","info":{"attrs":{"mtime":"2021-02-11T06:50:13.461392Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"68"}}},{"id":"1613026213.547780.c5e3283c","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:13.547780Z","info":{"attrs":{"mtime":"2021-02-11T06:50:13.417899Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"67"}}},{"id":"1613026213.600292.28f44133","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:13.600292Z","info":{"attrs":{"mtime":"2021-02-11T06:50:13.505702Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"69"}}},{"id":"1613026213.743774.650715a3","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:13.743774Z","info":{"attrs":{"mtime":"2021-02-11T06:50:13.637744Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"72"}}},{"id":"1613026213.744010.ad39db30","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:13.744010Z","info":{"attrs":{"mtime":"2021-02-11T06:50:13.681558Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"73"}}},{"id":"1613026213.783379.63aac8db","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:13.783379Z","info":{"attrs":{"mtime":"2021-02-11T06:50:13.549599Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"70"}}},{"id":"1613026213.824179.4fb12e54","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:13.824179Z","info":{"attrs":{"mtime":"2021-02-11T06:50:13.593793Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"71"}}},{"id":"1613026213.949250.82ffdd90","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:13.949250Z","info":{"attrs":{"mtime":"2021-02-11T06:50:13.905624Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"78"}}},{"id":"1613026213.952518.fc5b486c","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:13.952518Z","info":{"attrs":{"mtime":"2021-02-11T06:50:13.769782Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"75"}}},{"id":"1613026213.996281.5b329b35","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:13.996281Z","info":{"attrs":{"mtime":"2021-02-11T06:50:13.861569Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"77"}}},{"id":"1613026213.999008.a1827bf1","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:13.999008Z","info":{"attrs":{"mtime":"2021-02-11T06:50:13.725577Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"74"}}},{"id":"1613026214.032221.f71913c7","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:14.032221Z","info":{"attrs":{"mtime":"2021-02-11T06:50:13.817389Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"76"}}},{"id":"1613026214.144158.7fe91463","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:14.144158Z","info":{"attrs":{"mtime":"2021-02-11T06:50:13.993386Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"80"}}},{"id":"1613026214.186508.468b140e","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:14.186508Z","info":{"attrs":{"mtime":"2021-02-11T06:50:13.949509Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"79"}}},{"id":"1613026214.188421.40fcb94e","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:14.188421Z","info":{"attrs":{"mtime":"2021-02-11T06:50:14.082232Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"82"}}},{"id":"1613026214.228291.b234da37","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:14.228291Z","info":{"attrs":{"mtime":"2021-02-11T06:50:14.038253Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"81"}}},{"id":"1613026214.347705.6c55fd1b","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:14.347705Z","info":{"attrs":{"mtime":"2021-02-11T06:50:14.301735Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"87"}}},{"id":"1613026214.349675.572c6077","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:14.349675Z","info":{"attrs":{"mtime":"2021-02-11T06:50:14.258177Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"86"}}},{"id":"1613026214.350699.f2699be8","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:14.350699Z","info":{"attrs":{"mtime":"2021-02-11T06:50:14.213265Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"85"}}},{"id":"1613026214.391137.09ab1570","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:14.391137Z","info":{"attrs":{"mtime":"2021-02-11T06:50:14.125580Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"83"}}},{"id":"1613026214.440280.89992008","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:14.440280Z","info":{"attrs":{"mtime":"2021-02-11T06:50:14.169505Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"84"}}},{"id":"1613026214.541811.452ddfc3","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:14.541811Z","info":{"attrs":{"mtime":"2021-02-11T06:50:14.345650Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"88"}}},{"id":"1613026214.544885.397fcd8d","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:14.544885Z","info":{"attrs":{"mtime":"2021-02-11T06:50:14.481841Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"91"}}},{"id":"1613026214.592289.69f54826","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:14.592289Z","info":{"attrs":{"mtime":"2021-02-11T06:50:14.389778Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"89"}}},{"id":"1613026214.596293.f732eb3e","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:14.596293Z","info":{"attrs":{"mtime":"2021-02-11T06:50:14.437581Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"90"}}},{"id":"1613026214.747454.000ad945","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:14.747454Z","info":{"attrs":{"mtime":"2021-02-11T06:50:14.701663Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"96"}}},{"id":"1613026214.748043.2449772a","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:14.748043Z","info":{"attrs":{"mtime":"2021-02-11T06:50:14.526258Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"92"}}},{"id":"1613026214.791210.72f07f07","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:14.791210Z","info":{"attrs":{"mtime":"2021-02-11T06:50:14.658246Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"95"}}},{"id":"1613026214.794965.9133d910","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:14.794965Z","info":{"attrs":{"mtime":"2021-02-11T06:50:14.613443Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"94"}}},{"id":"1613026214.804879.92b898f9","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:14.804879Z","info":{"attrs":{"mtime":"2021-02-11T06:50:14.569290Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"93"}}},{"id":"1613026214.842009.c68bd88c","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:14.842009Z","info":{"attrs":{"mtime":"2021-02-11T06:50:14.754675Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"97"}}},{"id":"1613026214.941313.203b4715","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:14.941313Z","info":{"attrs":{"mtime":"2021-02-11T06:50:14.800204Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"98"}}}]} 2021-02-11T06:50:16.051 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['subscription', 'pull', '--uid', 'rgw-multisite-test-user', '--subscription', 'xscaju-82_sub', '--marker', 'xscaju-82_sub/1613026214.941313.203b4715', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:50:16.051 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'subscription', 'pull', '--uid', 'rgw-multisite-test-user', '--subscription', 'xscaju-82_sub', '--marker', 'xscaju-82_sub/1613026214.941313.203b4715', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T06:50:16.051 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 subscription pull --uid rgw-multisite-test-user --subscription xscaju-82_sub --marker xscaju-82_sub/1613026214.941313.203b4715 --rgw-zone test-zone4 --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T06:50:16.132 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:50:16.132 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:50:16.398 INFO:teuthology.orchestra.run.smithi172.stdout:{"next_marker":"","is_truncated":"false","events":[{"id":"1613026214.942197.e957d152","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:14.942197Z","info":{"attrs":{"mtime":"2021-02-11T06:50:14.901598Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"100"}}},{"id":"1613026214.988666.64fc7970","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:14.988666Z","info":{"attrs":{"mtime":"2021-02-11T06:50:14.853588Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"99"}}},{"id":"1613026215.144148.ee24834a","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:15.144148Z","info":{"attrs":{"mtime":"2021-02-11T06:50:15.088102Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"104"}}},{"id":"1613026215.192274.95f262fc","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:15.192274Z","info":{"attrs":{"mtime":"2021-02-11T06:50:15.040503Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"103"}}},{"id":"1613026215.192630.a137fa87","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:15.192630Z","info":{"attrs":{"mtime":"2021-02-11T06:50:14.995729Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"102"}}},{"id":"1613026215.195229.7d094034","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:15.195229Z","info":{"attrs":{"mtime":"2021-02-11T06:50:14.948833Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"101"}}},{"id":"1613026215.345594.53fd5851","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:15.345594Z","info":{"attrs":{"mtime":"2021-02-11T06:50:15.271012Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"108"}}},{"id":"1613026215.347406.46272d79","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:15.347406Z","info":{"attrs":{"mtime":"2021-02-11T06:50:15.226688Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"107"}}},{"id":"1613026215.347637.4f7a0e52","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:15.347637Z","info":{"attrs":{"mtime":"2021-02-11T06:50:15.136256Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"105"}}},{"id":"1613026215.347733.ef051acb","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:15.347733Z","info":{"attrs":{"mtime":"2021-02-11T06:50:15.182350Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"106"}}},{"id":"1613026215.432259.50a283f0","event":"OBJECT_CREATE","timestamp":"2021-02-11T06:50:15.432259Z","info":{"attrs":{"mtime":"2021-02-11T06:50:15.315852Z"},"bucket":{"bucket_id":"5f91a5b1-d54f-4a3d-a100-b2911a08a2e7.5513.30","name":"xscaju-82","tenant":""},"key":{"instance":"","name":"109"}}}]} 2021-02-11T06:50:16.427 INFO:tasks.rgw_multisite_tests:test radosgw-admin commands ... FAIL 2021-02-11T06:50:16.428 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:50:16.428 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:50:16.428 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:50:16.468 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:50:16.468 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:50:16.743 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=0d061ff0-ede8-4baa-beef-d975d317d505 2021-02-11T06:50:16.786 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613026209.451303_381.1","last_update":"2021-02-11T06:50:09.451303Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:50:16.786 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T06:50:16.786 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:50:16.786 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:50:16.786 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:50:16.823 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:50:16.823 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:50:17.094 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026209.451303_381.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:50:09.451303Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:50:17.094 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T06:50:17.095 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T06:50:17.095 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:50:17.095 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:50:17.095 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:50:17.135 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:50:17.135 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:50:17.417 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026209.451303_381.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:50:09.451303Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:50:17.418 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T06:50:17.418 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:50:17.418 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:50:17.418 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:50:17.418 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:50:17.458 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:50:17.458 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:50:17.714 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026209.451303_381.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:50:09.451303Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:50:17.714 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:50:17.714 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:50:17.714 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:50:17.715 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:50:17.755 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:50:17.755 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:50:18.026 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=0d061ff0-ede8-4baa-beef-d975d317d505 2021-02-11T06:50:18.065 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613026209.451303_381.1","last_update":"2021-02-11T06:50:09.451303Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:50:18.065 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:50:18.065 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:50:18.066 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:50:18.066 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:50:18.106 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:50:18.107 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:50:18.369 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026209.451303_381.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:50:09.451303Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:50:18.369 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:50:18.534 INFO:tasks.rgw.c2.client.0.smithi172.stdout:2021-02-11T06:50:18.531+0000 7fb3cf7fe700 -1 data sync zone:5f91a5b1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T06:50:18.534 INFO:tasks.rgw.c2.client.0.smithi172.stdout:2021-02-11T06:50:18.531+0000 7fb3c2ff5700 -1 data sync zone:5f91a5b1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T06:50:18.534 INFO:tasks.rgw.c2.client.1.smithi172.stdout:2021-02-11T06:50:18.531+0000 7f2bbefed700 -1 data sync zone:5f91a5b1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T06:50:18.534 INFO:tasks.rgw.c2.client.1.smithi172.stdout:2021-02-11T06:50:18.531+0000 7f2bc3ff7700 -1 data sync zone:5f91a5b1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T06:50:18.535 INFO:tasks.rgw.c1.client.1.smithi061.stdout:2021-02-11T06:50:18.530+0000 7f701b7ee700 -1 data sync zone:5f91a5b1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T06:50:18.535 INFO:tasks.rgw.c1.client.1.smithi061.stdout:2021-02-11T06:50:18.530+0000 7f7023fff700 -1 data sync zone:5f91a5b1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T06:50:18.883 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-83', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:50:18.883 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-83', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:50:18.883 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-83 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:50:18.927 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:50:18.928 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:50:19.188 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:50:19.183+0000 7f78fb264080 1 bucket sync caught up with source: 2021-02-11T06:50:19.188 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000002.98.5, , 00000000001.140.5, 00000000001.65.5, 00000000001.80.5, , 00000000001.104.5, 00000000001.214.5, 00000000001.63.5, 00000000002.77.5] 2021-02-11T06:50:19.188 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000002.98.5, , 00000000001.140.5, 00000000001.65.5, 00000000001.80.5, , 00000000001.104.5, 00000000001.214.5, 00000000001.63.5, 00000000002.77.5] 2021-02-11T06:50:19.188 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:50:19.183+0000 7f78fb264080 0 bucket checkpoint complete 2021-02-11T06:50:19.647 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-83', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:50:19.647 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-83', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:50:19.647 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-83 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:50:19.701 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:50:19.701 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:50:19.945 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:50:19.943+0000 7f002c9a0080 1 bucket sync caught up with source: 2021-02-11T06:50:19.946 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [00000000001.71.5, 00000000003.100.5, , 00000000002.142.5, 00000000001.65.5, 00000000003.84.5, , 00000000003.108.5, 00000000002.216.5, 00000000002.67.5, 00000000003.79.5] 2021-02-11T06:50:19.946 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [00000000001.71.5, 00000000003.100.5, , 00000000002.142.5, 00000000001.65.5, 00000000003.84.5, , 00000000003.108.5, 00000000002.216.5, 00000000002.67.5, 00000000003.79.5] 2021-02-11T06:50:19.946 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:50:19.943+0000 7f002c9a0080 0 bucket checkpoint complete 2021-02-11T06:50:19.981 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "11" type: "OBJECT_CREATE" 2021-02-11T06:50:19.981 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "10" type: "OBJECT_CREATE" 2021-02-11T06:50:19.981 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "12" type: "OBJECT_CREATE" 2021-02-11T06:50:19.981 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "13" type: "OBJECT_CREATE" 2021-02-11T06:50:19.981 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "15" type: "OBJECT_CREATE" 2021-02-11T06:50:19.981 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "14" type: "OBJECT_CREATE" 2021-02-11T06:50:19.981 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "16" type: "OBJECT_CREATE" 2021-02-11T06:50:19.981 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "17" type: "OBJECT_CREATE" 2021-02-11T06:50:19.981 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "18" type: "OBJECT_CREATE" 2021-02-11T06:50:19.981 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "19" type: "OBJECT_CREATE" 2021-02-11T06:50:20.361 INFO:tasks.rgw_multisite_tests:test that events are only sent on incremental sync ... ok 2021-02-11T06:50:20.361 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:50:20.362 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:50:20.362 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:50:20.402 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:50:20.402 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:50:20.616 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=0d061ff0-ede8-4baa-beef-d975d317d505 2021-02-11T06:50:41.806 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613026220.352580_397.1","last_update":"2021-02-11T06:50:20.352580Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:50:41.807 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T06:50:41.807 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:50:41.807 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:50:41.807 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:50:41.849 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:50:41.849 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:50:42.118 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026220.352580_397.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:50:20.352580Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:50:42.118 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T06:50:42.118 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T06:50:42.119 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:50:42.119 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:50:42.119 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:50:42.159 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:50:42.159 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:50:42.453 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026220.352580_397.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:50:20.352580Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:50:42.453 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T06:50:42.453 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:50:42.453 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:50:42.453 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:50:42.454 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:50:42.498 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:50:42.498 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:50:42.752 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026220.352580_397.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:50:20.352580Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:50:42.753 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:50:42.753 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:50:42.753 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:50:42.753 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:50:42.790 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:50:42.790 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:50:43.032 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=0d061ff0-ede8-4baa-beef-d975d317d505 2021-02-11T06:50:43.065 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613026220.352580_397.1","last_update":"2021-02-11T06:50:20.352580Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:50:43.065 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:50:43.066 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:50:43.066 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:50:43.066 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:50:43.106 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:50:43.106 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:50:43.365 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026220.352580_397.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:50:20.352580Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:50:43.365 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:50:43.443 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:50:43.444 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:50:43.444 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:50:43.483 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:50:43.483 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:50:43.724 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=0d061ff0-ede8-4baa-beef-d975d317d505 2021-02-11T06:50:48.137 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613026243.439219_403.1","last_update":"2021-02-11T06:50:43.439219Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:50:48.137 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:50:48.137 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:50:48.138 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:50:48.138 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:50:48.178 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:50:48.178 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:50:48.457 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026243.439219_403.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:50:43.439219Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:50:48.457 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:50:48.457 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-84', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:50:48.458 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-84', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:50:48.458 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-84 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:50:48.545 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:50:48.545 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:50:48.789 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:50:48.786+0000 7f27ba8e3080 1 bucket sync caught up with empty source 2021-02-11T06:50:48.789 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:50:48.786+0000 7f27ba8e3080 0 bucket checkpoint complete 2021-02-11T06:50:49.307 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-84', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:50:49.307 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-84', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:50:49.307 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-84 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:50:49.356 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:50:49.357 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:50:49.595 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:50:49.590+0000 7f18c6e58080 1 waiting for incremental sync to catch up: 2021-02-11T06:50:49.596 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000002.78.5, , 00000000001.131.5, , 00000000001.105.5, , , 00000000001.97.5, 00000000001.122.5, 00000000002.105.5] 2021-02-11T06:50:49.596 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000002.78.5, , 00000000001.131.5, 00000000001.123.5, 00000000001.105.5, , 00000000001.98.5, 00000000001.97.5, 00000000001.122.5, 00000000002.105.5] 2021-02-11T06:50:54.601 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:50:54.594+0000 7f18c6e58080 1 waiting for incremental sync to catch up: 2021-02-11T06:50:54.601 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000002.78.5, , 00000000001.131.5, , 00000000001.105.5, , 00000000001.98.5, 00000000001.97.5, 00000000001.122.5, 00000000002.105.5] 2021-02-11T06:50:54.601 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000002.78.5, , 00000000001.131.5, 00000000001.123.5, 00000000001.105.5, , 00000000001.98.5, 00000000001.97.5, 00000000001.122.5, 00000000002.105.5] 2021-02-11T06:50:59.603 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:50:59.598+0000 7f18c6e58080 1 waiting for incremental sync to catch up: 2021-02-11T06:50:59.603 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000002.78.5, , 00000000001.131.5, , 00000000001.105.5, , 00000000001.98.5, 00000000001.97.5, 00000000001.122.5, 00000000002.105.5] 2021-02-11T06:50:59.604 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000002.78.5, , 00000000001.131.5, 00000000001.123.5, 00000000001.105.5, , 00000000001.98.5, 00000000001.97.5, 00000000001.122.5, 00000000002.105.5] 2021-02-11T06:51:04.608 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:51:04.602+0000 7f18c6e58080 1 waiting for incremental sync to catch up: 2021-02-11T06:51:23.733 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000002.78.5, , 00000000001.131.5, , 00000000001.105.5, , 00000000001.98.5, 00000000001.97.5, 00000000001.122.5, 00000000002.105.5] 2021-02-11T06:51:23.734 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000002.78.5, , 00000000001.131.5, 00000000001.123.5, 00000000001.105.5, , 00000000001.98.5, 00000000001.97.5, 00000000001.122.5, 00000000002.105.5] 2021-02-11T06:51:23.736 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:51:09.606+0000 7f18c6e58080 1 bucket sync caught up with source: 2021-02-11T06:51:23.737 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000002.78.5, , 00000000001.131.5, 00000000001.123.5, 00000000001.105.5, , 00000000001.98.5, 00000000001.97.5, 00000000001.122.5, 00000000002.105.5] 2021-02-11T06:51:23.737 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000002.78.5, , 00000000001.131.5, 00000000001.123.5, 00000000001.105.5, , 00000000001.98.5, 00000000001.97.5, 00000000001.122.5, 00000000002.105.5] 2021-02-11T06:51:23.737 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:51:09.606+0000 7f18c6e58080 0 bucket checkpoint complete 2021-02-11T06:51:23.749 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "0" type: "OBJECT_CREATE" 2021-02-11T06:51:23.749 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "1" type: "OBJECT_CREATE" 2021-02-11T06:51:23.749 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "5" type: "OBJECT_CREATE" 2021-02-11T06:51:23.750 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "6" type: "OBJECT_CREATE" 2021-02-11T06:51:23.750 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "9" type: "OBJECT_CREATE" 2021-02-11T06:51:23.750 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "8" type: "OBJECT_CREATE" 2021-02-11T06:51:23.750 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "4" type: "OBJECT_CREATE" 2021-02-11T06:51:23.750 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "3" type: "OBJECT_CREATE" 2021-02-11T06:51:23.750 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "2" type: "OBJECT_CREATE" 2021-02-11T06:51:23.751 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "7" type: "OBJECT_CREATE" 2021-02-11T06:51:23.763 DEBUG:tasks.rgw_multi.tests_ps:superfluous events are found 2021-02-11T06:51:23.776 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "0" type: "OBJECT_CREATE" 2021-02-11T06:51:23.776 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "1" type: "OBJECT_CREATE" 2021-02-11T06:51:23.776 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "5" type: "OBJECT_CREATE" 2021-02-11T06:51:23.776 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "6" type: "OBJECT_CREATE" 2021-02-11T06:51:23.777 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "9" type: "OBJECT_CREATE" 2021-02-11T06:51:23.777 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "8" type: "OBJECT_CREATE" 2021-02-11T06:51:23.777 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "4" type: "OBJECT_CREATE" 2021-02-11T06:51:23.777 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "3" type: "OBJECT_CREATE" 2021-02-11T06:51:23.777 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "2" type: "OBJECT_CREATE" 2021-02-11T06:51:23.777 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "7" type: "OBJECT_CREATE" 2021-02-11T06:51:23.777 DEBUG:tasks.rgw_multi.tests_ps:superfluous events are found 2021-02-11T06:51:23.891 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-84', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:51:23.891 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-84', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:51:23.892 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-84 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:51:23.939 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:51:23.940 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:51:24.224 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:51:24.217+0000 7ff8de018080 1 bucket sync caught up with source: 2021-02-11T06:51:24.224 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000004.82.5, , 00000000002.133.5, 00000000002.125.5, 00000000002.107.5, , 00000000002.100.5, 00000000002.99.5, 00000000002.124.5, 00000000004.109.5] 2021-02-11T06:51:24.224 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000004.82.5, , 00000000002.133.5, 00000000002.125.5, 00000000002.107.5, , 00000000002.100.5, 00000000002.99.5, 00000000002.124.5, 00000000004.109.5] 2021-02-11T06:51:24.224 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:51:24.217+0000 7ff8de018080 0 bucket checkpoint complete 2021-02-11T06:51:24.255 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_DELETE) synced 2021-02-11T06:51:24.263 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "0" type: "OBJECT_CREATE" 2021-02-11T06:51:24.263 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "1" type: "OBJECT_CREATE" 2021-02-11T06:51:24.263 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "5" type: "OBJECT_CREATE" 2021-02-11T06:51:24.263 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "6" type: "OBJECT_CREATE" 2021-02-11T06:51:24.263 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "9" type: "OBJECT_CREATE" 2021-02-11T06:51:24.263 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "8" type: "OBJECT_CREATE" 2021-02-11T06:51:24.263 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "4" type: "OBJECT_CREATE" 2021-02-11T06:51:24.263 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "3" type: "OBJECT_CREATE" 2021-02-11T06:51:24.264 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "2" type: "OBJECT_CREATE" 2021-02-11T06:51:24.264 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "7" type: "OBJECT_CREATE" 2021-02-11T06:51:24.264 DEBUG:tasks.rgw_multi.tests_ps:superfluous events are found 2021-02-11T06:51:24.272 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_DELETE): objname: "1" type: "OBJECT_DELETE" 2021-02-11T06:51:24.272 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_DELETE): objname: "7" type: "OBJECT_DELETE" 2021-02-11T06:51:24.272 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_DELETE): objname: "5" type: "OBJECT_DELETE" 2021-02-11T06:51:24.272 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_DELETE): objname: "9" type: "OBJECT_DELETE" 2021-02-11T06:51:24.272 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_DELETE): objname: "2" type: "OBJECT_DELETE" 2021-02-11T06:51:24.273 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_DELETE): objname: "6" type: "OBJECT_DELETE" 2021-02-11T06:51:24.273 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_DELETE): objname: "0" type: "OBJECT_DELETE" 2021-02-11T06:51:24.273 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_DELETE): objname: "4" type: "OBJECT_DELETE" 2021-02-11T06:51:24.273 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_DELETE): objname: "8" type: "OBJECT_DELETE" 2021-02-11T06:51:24.273 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_DELETE): objname: "3" type: "OBJECT_DELETE" 2021-02-11T06:51:24.273 DEBUG:tasks.rgw_multi.tests_ps:superfluous events are found 2021-02-11T06:51:24.282 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "0" type: "OBJECT_CREATE" 2021-02-11T06:51:24.282 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "1" type: "OBJECT_CREATE" 2021-02-11T06:51:24.282 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "5" type: "OBJECT_CREATE" 2021-02-11T06:51:24.282 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "6" type: "OBJECT_CREATE" 2021-02-11T06:51:24.282 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "9" type: "OBJECT_CREATE" 2021-02-11T06:51:24.282 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "8" type: "OBJECT_CREATE" 2021-02-11T06:51:24.282 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "4" type: "OBJECT_CREATE" 2021-02-11T06:51:24.282 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "3" type: "OBJECT_CREATE" 2021-02-11T06:51:24.282 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "2" type: "OBJECT_CREATE" 2021-02-11T06:51:24.283 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "7" type: "OBJECT_CREATE" 2021-02-11T06:51:24.283 DEBUG:tasks.rgw_multi.tests_ps:superfluous events are found 2021-02-11T06:51:24.473 INFO:tasks.rgw_multisite_tests:test subscriptions for different events ... ok 2021-02-11T06:51:24.474 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:51:24.474 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:51:24.474 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:51:24.514 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:51:24.515 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:51:24.759 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=0d061ff0-ede8-4baa-beef-d975d317d505 2021-02-11T06:51:24.798 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613026284.459572_409.1","last_update":"2021-02-11T06:51:24.459572Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:51:24.799 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T06:51:24.799 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:51:24.799 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:51:24.799 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:51:24.841 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:51:24.841 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:51:25.121 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026284.459572_409.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:51:24.459572Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:51:25.122 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T06:51:25.122 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T06:51:25.122 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:51:25.122 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:51:25.122 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:51:25.163 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:51:25.164 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:51:25.433 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026284.459572_409.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:51:24.459572Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:51:25.434 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T06:51:25.434 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:51:25.434 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:51:25.434 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:51:25.434 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:51:25.480 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:51:26.205 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:51:26.207 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026284.459572_409.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:51:24.459572Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:51:26.207 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:51:26.208 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:51:26.208 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:51:26.208 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:51:26.244 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:51:26.244 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:51:26.494 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=0d061ff0-ede8-4baa-beef-d975d317d505 2021-02-11T06:51:26.533 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613026284.459572_409.1","last_update":"2021-02-11T06:51:24.459572Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:51:26.534 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:51:26.534 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:51:26.534 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:51:26.534 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:51:26.575 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:51:26.575 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:51:26.842 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026284.459572_409.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:51:24.459572Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:51:26.842 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:51:26.915 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:51:26.915 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:51:26.915 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:51:26.958 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:51:26.958 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:51:27.206 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=0d061ff0-ede8-4baa-beef-d975d317d505 2021-02-11T06:51:27.243 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613026286.909682_413.1","last_update":"2021-02-11T06:51:26.909682Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:51:27.243 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:51:27.244 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:51:27.244 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:51:27.244 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:51:27.285 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:51:27.285 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:51:27.547 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026286.909682_413.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:51:26.909682Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:51:27.547 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:51:32.031 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-85', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:51:48.227 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-85', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:51:48.228 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-85 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:51:48.281 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:51:48.281 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:51:48.519 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:51:48.513+0000 7fb49b3d3080 1 bucket sync caught up with source: 2021-02-11T06:51:48.519 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [00000000010.120.5, 00000000010.241.5, 00000000009.131.5, 00000000010.132.5, 00000000007.83.5, 00000000007.140.5, 00000000008.144.5, 00000000010.139.5, 00000000010.140.5, 00000000010.136.5, 00000000009.98.5] 2021-02-11T06:51:48.519 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [00000000010.120.5, 00000000010.241.5, 00000000009.131.5, 00000000010.132.5, 00000000007.83.5, 00000000007.140.5, 00000000008.144.5, 00000000010.139.5, 00000000010.140.5, 00000000010.136.5, 00000000009.98.5] 2021-02-11T06:51:48.519 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:51:48.513+0000 7fb49b3d3080 0 bucket checkpoint complete 2021-02-11T06:51:48.560 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "0" type: "OBJECT_CREATE" 2021-02-11T06:51:48.560 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "4" type: "OBJECT_CREATE" 2021-02-11T06:51:48.561 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "3" type: "OBJECT_CREATE" 2021-02-11T06:51:48.561 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "1" type: "OBJECT_CREATE" 2021-02-11T06:51:48.561 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "6" type: "OBJECT_CREATE" 2021-02-11T06:51:48.561 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "5" type: "OBJECT_CREATE" 2021-02-11T06:51:48.561 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "9" type: "OBJECT_CREATE" 2021-02-11T06:51:48.561 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "7" type: "OBJECT_CREATE" 2021-02-11T06:51:48.561 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "8" type: "OBJECT_CREATE" 2021-02-11T06:51:48.561 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "13" type: "OBJECT_CREATE" 2021-02-11T06:51:48.561 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "12" type: "OBJECT_CREATE" 2021-02-11T06:51:48.561 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "11" type: "OBJECT_CREATE" 2021-02-11T06:51:48.561 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "2" type: "OBJECT_CREATE" 2021-02-11T06:51:48.561 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "15" type: "OBJECT_CREATE" 2021-02-11T06:51:48.561 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "16" type: "OBJECT_CREATE" 2021-02-11T06:51:48.569 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "17" type: "OBJECT_CREATE" 2021-02-11T06:51:48.569 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "14" type: "OBJECT_CREATE" 2021-02-11T06:51:48.569 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "18" type: "OBJECT_CREATE" 2021-02-11T06:51:48.569 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "21" type: "OBJECT_CREATE" 2021-02-11T06:51:48.569 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "19" type: "OBJECT_CREATE" 2021-02-11T06:51:48.569 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "20" type: "OBJECT_CREATE" 2021-02-11T06:51:48.569 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "22" type: "OBJECT_CREATE" 2021-02-11T06:51:48.569 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "10" type: "OBJECT_CREATE" 2021-02-11T06:51:48.569 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "25" type: "OBJECT_CREATE" 2021-02-11T06:51:48.569 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "26" type: "OBJECT_CREATE" 2021-02-11T06:51:48.569 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "24" type: "OBJECT_CREATE" 2021-02-11T06:51:48.569 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "28" type: "OBJECT_CREATE" 2021-02-11T06:51:48.569 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "30" type: "OBJECT_CREATE" 2021-02-11T06:51:48.569 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "29" type: "OBJECT_CREATE" 2021-02-11T06:51:48.570 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "31" type: "OBJECT_CREATE" 2021-02-11T06:51:48.577 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "34" type: "OBJECT_CREATE" 2021-02-11T06:51:48.577 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "35" type: "OBJECT_CREATE" 2021-02-11T06:51:48.577 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "33" type: "OBJECT_CREATE" 2021-02-11T06:51:48.577 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "32" type: "OBJECT_CREATE" 2021-02-11T06:51:48.577 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "36" type: "OBJECT_CREATE" 2021-02-11T06:51:48.577 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "23" type: "OBJECT_CREATE" 2021-02-11T06:51:48.578 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "38" type: "OBJECT_CREATE" 2021-02-11T06:51:48.578 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "37" type: "OBJECT_CREATE" 2021-02-11T06:51:48.578 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "27" type: "OBJECT_CREATE" 2021-02-11T06:51:48.578 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "39" type: "OBJECT_CREATE" 2021-02-11T06:51:48.578 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "40" type: "OBJECT_CREATE" 2021-02-11T06:51:48.578 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "41" type: "OBJECT_CREATE" 2021-02-11T06:51:48.578 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "43" type: "OBJECT_CREATE" 2021-02-11T06:51:48.578 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "42" type: "OBJECT_CREATE" 2021-02-11T06:51:48.578 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "44" type: "OBJECT_CREATE" 2021-02-11T06:51:48.586 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "46" type: "OBJECT_CREATE" 2021-02-11T06:51:48.586 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "49" type: "OBJECT_CREATE" 2021-02-11T06:51:48.586 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "48" type: "OBJECT_CREATE" 2021-02-11T06:51:48.586 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "45" type: "OBJECT_CREATE" 2021-02-11T06:51:48.586 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "47" type: "OBJECT_CREATE" 2021-02-11T06:51:48.586 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "51" type: "OBJECT_CREATE" 2021-02-11T06:51:48.586 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "50" type: "OBJECT_CREATE" 2021-02-11T06:51:48.586 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "53" type: "OBJECT_CREATE" 2021-02-11T06:51:48.586 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "54" type: "OBJECT_CREATE" 2021-02-11T06:51:48.586 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "52" type: "OBJECT_CREATE" 2021-02-11T06:51:48.587 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "57" type: "OBJECT_CREATE" 2021-02-11T06:51:48.587 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "56" type: "OBJECT_CREATE" 2021-02-11T06:51:48.587 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "55" type: "OBJECT_CREATE" 2021-02-11T06:51:48.587 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "58" type: "OBJECT_CREATE" 2021-02-11T06:51:48.587 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "61" type: "OBJECT_CREATE" 2021-02-11T06:51:48.594 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "60" type: "OBJECT_CREATE" 2021-02-11T06:51:48.594 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "62" type: "OBJECT_CREATE" 2021-02-11T06:51:48.594 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "59" type: "OBJECT_CREATE" 2021-02-11T06:51:48.595 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "63" type: "OBJECT_CREATE" 2021-02-11T06:51:48.595 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "67" type: "OBJECT_CREATE" 2021-02-11T06:51:48.595 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "65" type: "OBJECT_CREATE" 2021-02-11T06:51:48.595 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "64" type: "OBJECT_CREATE" 2021-02-11T06:51:48.595 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "66" type: "OBJECT_CREATE" 2021-02-11T06:51:48.595 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "70" type: "OBJECT_CREATE" 2021-02-11T06:51:48.595 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "68" type: "OBJECT_CREATE" 2021-02-11T06:51:48.595 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "69" type: "OBJECT_CREATE" 2021-02-11T06:51:48.595 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "71" type: "OBJECT_CREATE" 2021-02-11T06:51:48.595 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "73" type: "OBJECT_CREATE" 2021-02-11T06:51:48.595 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "75" type: "OBJECT_CREATE" 2021-02-11T06:51:48.595 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "74" type: "OBJECT_CREATE" 2021-02-11T06:51:48.602 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "76" type: "OBJECT_CREATE" 2021-02-11T06:51:48.602 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "72" type: "OBJECT_CREATE" 2021-02-11T06:51:48.602 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "79" type: "OBJECT_CREATE" 2021-02-11T06:51:48.602 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "80" type: "OBJECT_CREATE" 2021-02-11T06:51:48.602 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "78" type: "OBJECT_CREATE" 2021-02-11T06:51:48.603 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "77" type: "OBJECT_CREATE" 2021-02-11T06:51:48.603 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "84" type: "OBJECT_CREATE" 2021-02-11T06:51:48.603 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "82" type: "OBJECT_CREATE" 2021-02-11T06:51:48.603 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "81" type: "OBJECT_CREATE" 2021-02-11T06:51:48.603 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "85" type: "OBJECT_CREATE" 2021-02-11T06:51:48.603 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "83" type: "OBJECT_CREATE" 2021-02-11T06:51:48.603 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "88" type: "OBJECT_CREATE" 2021-02-11T06:51:48.603 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "89" type: "OBJECT_CREATE" 2021-02-11T06:51:48.603 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "87" type: "OBJECT_CREATE" 2021-02-11T06:51:48.603 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "86" type: "OBJECT_CREATE" 2021-02-11T06:51:48.610 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "93" type: "OBJECT_CREATE" 2021-02-11T06:51:48.610 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "90" type: "OBJECT_CREATE" 2021-02-11T06:51:48.611 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "94" type: "OBJECT_CREATE" 2021-02-11T06:51:48.611 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "95" type: "OBJECT_CREATE" 2021-02-11T06:51:48.611 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "92" type: "OBJECT_CREATE" 2021-02-11T06:51:48.611 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "91" type: "OBJECT_CREATE" 2021-02-11T06:51:48.611 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "96" type: "OBJECT_CREATE" 2021-02-11T06:51:48.611 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "97" type: "OBJECT_CREATE" 2021-02-11T06:51:48.611 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "98" type: "OBJECT_CREATE" 2021-02-11T06:51:48.611 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "99" type: "OBJECT_CREATE" 2021-02-11T06:51:48.631 DEBUG:tasks.rgw_multi.tests_ps:superfluous events are found 2021-02-11T06:51:50.109 INFO:tasks.rgw_multisite_tests:test incremental fetching of events from a subscription ... ok 2021-02-11T06:51:50.110 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:51:50.110 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:51:50.110 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:51:50.156 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:51:50.156 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:51:50.370 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=0d061ff0-ede8-4baa-beef-d975d317d505 2021-02-11T06:51:50.405 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613026310.099809_415.1","last_update":"2021-02-11T06:51:50.099809Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:51:50.405 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T06:51:50.406 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:51:50.406 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:51:50.406 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:51:50.446 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:51:50.446 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:51:50.848 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026310.099809_415.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:51:50.099809Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:51:50.848 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T06:51:50.848 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T06:51:50.849 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:51:50.849 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:51:50.849 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:51:50.888 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:51:50.888 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:51:51.167 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026310.099809_415.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:51:50.099809Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:51:51.167 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T06:51:51.167 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:51:51.167 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:51:51.167 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:51:51.168 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:51:51.214 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:51:51.214 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:51:51.484 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026310.099809_415.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:51:50.099809Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:51:51.484 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:51:51.484 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:51:51.484 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:51:51.485 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:51:51.530 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:51:51.530 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:51:51.788 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=0d061ff0-ede8-4baa-beef-d975d317d505 2021-02-11T06:51:51.830 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613026310.099809_415.1","last_update":"2021-02-11T06:51:50.099809Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:51:51.830 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:51:51.830 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:51:51.831 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:51:51.831 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:51:51.872 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:51:51.872 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:51:52.142 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026310.099809_415.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:51:50.099809Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:51:52.143 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:51:52.200 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:51:52.201 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:51:52.201 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:51:52.237 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:51:52.237 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:51:52.494 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=0d061ff0-ede8-4baa-beef-d975d317d505 2021-02-11T06:51:52.531 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613026312.195917_419.1","last_update":"2021-02-11T06:51:52.195917Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:51:52.531 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:51:52.531 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:51:52.531 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:51:52.532 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:51:52.573 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:51:52.573 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:51:52.839 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026312.195917_419.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:51:52.195917Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:51:52.839 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:51:53.283 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-86', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:51:57.293 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-86', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:51:57.294 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-86 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:51:57.340 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:51:57.341 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:51:57.597 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:51:57.593+0000 7f2a44d65080 1 bucket sync caught up with source: 2021-02-11T06:51:57.598 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000002.181.5, , 00000000001.111.5, 00000000001.112.5, 00000000001.143.5, , 00000000001.131.5, 00000000001.152.5, 00000000001.101.5, 00000000002.105.5] 2021-02-11T06:51:57.598 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000002.181.5, , 00000000001.111.5, 00000000001.112.5, 00000000001.143.5, , 00000000001.131.5, 00000000001.152.5, 00000000001.101.5, 00000000002.105.5] 2021-02-11T06:51:57.598 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:51:57.593+0000 7f2a44d65080 0 bucket checkpoint complete 2021-02-11T06:51:57.640 DEBUG:tasks.rgw_multi.tests_ps:Event (before ack) id: "1613026313.050045.dd9f72d4" 2021-02-11T06:51:57.640 DEBUG:tasks.rgw_multi.tests_ps:Event (before ack) id: "1613026313.095057.3742cf88" 2021-02-11T06:51:57.640 DEBUG:tasks.rgw_multi.tests_ps:Event (before ack) id: "1613026313.144204.aaf40142" 2021-02-11T06:51:57.641 DEBUG:tasks.rgw_multi.tests_ps:Event (before ack) id: "1613026313.216071.d575ccbf" 2021-02-11T06:51:57.641 DEBUG:tasks.rgw_multi.tests_ps:Event (before ack) id: "1613026313.293267.8b2b0f46" 2021-02-11T06:51:57.641 DEBUG:tasks.rgw_multi.tests_ps:Event (before ack) id: "1613026313.299081.124610a0" 2021-02-11T06:51:57.641 DEBUG:tasks.rgw_multi.tests_ps:Event (before ack) id: "1613026313.338950.1e746d47" 2021-02-11T06:51:57.641 DEBUG:tasks.rgw_multi.tests_ps:Event (before ack) id: "1613026313.340191.94aae5f0" 2021-02-11T06:51:57.641 DEBUG:tasks.rgw_multi.tests_ps:Event (before ack) id: "1613026313.487205.4056ec1d" 2021-02-11T06:51:57.641 DEBUG:tasks.rgw_multi.tests_ps:Event (before ack) id: "1613026313.489349.f953b964" 2021-02-11T06:51:57.652 DEBUG:tasks.rgw_multi.tests_ps:superfluous events are found 2021-02-11T06:51:57.708 DEBUG:tasks.rgw_multi.tests_ps:Event (after ack) id: "1613026313.299081.124610a0" 2021-02-11T06:51:57.708 DEBUG:tasks.rgw_multi.tests_ps:Event (after ack) id: "1613026313.338950.1e746d47" 2021-02-11T06:51:57.708 DEBUG:tasks.rgw_multi.tests_ps:Event (after ack) id: "1613026313.340191.94aae5f0" 2021-02-11T06:51:57.708 DEBUG:tasks.rgw_multi.tests_ps:Event (after ack) id: "1613026313.487205.4056ec1d" 2021-02-11T06:51:57.708 DEBUG:tasks.rgw_multi.tests_ps:Event (after ack) id: "1613026313.489349.f953b964" 2021-02-11T06:51:57.963 INFO:tasks.rgw_multisite_tests:test acking of some events in a subscription ... ok 2021-02-11T06:51:57.964 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:51:57.964 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:51:57.965 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:51:58.015 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:51:58.016 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:51:58.244 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=0d061ff0-ede8-4baa-beef-d975d317d505 2021-02-11T06:51:58.272 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613026317.954332_421.1","last_update":"2021-02-11T06:51:57.954332Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:51:58.272 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T06:51:58.272 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:51:58.273 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:51:58.273 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:51:58.320 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:51:58.320 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:51:58.604 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026317.954332_421.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:51:57.954332Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:51:58.604 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T06:51:58.604 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T06:51:58.604 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:51:58.605 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:51:58.605 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:51:58.647 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:51:58.648 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:51:58.945 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026317.954332_421.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:51:57.954332Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:51:58.945 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T06:51:58.945 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:51:58.945 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:51:58.945 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:51:58.945 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:51:58.986 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:51:58.986 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:51:59.262 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026317.954332_421.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:51:57.954332Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:51:59.262 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:51:59.262 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:51:59.263 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:51:59.263 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:51:59.302 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:51:59.303 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:51:59.562 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=0d061ff0-ede8-4baa-beef-d975d317d505 2021-02-11T06:51:59.605 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613026317.954332_421.1","last_update":"2021-02-11T06:51:57.954332Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:51:59.606 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:51:59.606 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:51:59.606 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:51:59.606 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:51:59.648 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:51:59.648 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:51:59.923 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026317.954332_421.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:51:57.954332Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:51:59.923 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:51:59.988 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:51:59.988 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:51:59.988 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:52:00.042 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:52:00.043 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:52:00.274 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=0d061ff0-ede8-4baa-beef-d975d317d505 2021-02-11T06:52:00.320 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613026319.983022_425.1","last_update":"2021-02-11T06:51:59.983022Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:52:00.320 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:52:00.321 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:52:00.321 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:52:00.321 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:52:00.364 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:52:00.364 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:52:00.642 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026319.983022_425.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:51:59.983022Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:52:00.642 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:52:00.975 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-87', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:52:00.975 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-87', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:52:00.975 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-87 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:52:01.020 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:52:01.020 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:52:01.244 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:52:01.241+0000 7f0a11f6f080 1 bucket sync caught up with source: 2021-02-11T06:52:01.244 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [00000000001.183.5, , , , , , 00000000001.157.5, , , 00000000005.123.5, ] 2021-02-11T06:52:01.245 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [00000000001.183.5, , , , , , 00000000001.157.5, , , 00000000005.123.5, ] 2021-02-11T06:52:01.245 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:52:01.241+0000 7f0a11f6f080 0 bucket checkpoint complete 2021-02-11T06:52:01.279 DEBUG:tasks.rgw_multi.tests_ps:Event key: "put" type: "OBJECT_CREATE" 2021-02-11T06:52:01.279 DEBUG:tasks.rgw_multi.tests_ps:Event key: "copy" type: "OBJECT_CREATE" 2021-02-11T06:52:01.279 DEBUG:tasks.rgw_multi.tests_ps:Event key: "multipart" type: "OBJECT_CREATE" 2021-02-11T06:52:01.438 INFO:tasks.rgw_multisite_tests:test object creation notifications in using put/copy/post ... ok 2021-02-11T06:52:01.438 INFO:tasks.rgw_multisite_tests:test object creation s3 notifications in using put/copy/post on master ... ok 2021-02-11T06:52:01.438 INFO:tasks.rgw_multisite_tests:test multipart object upload on master ... ok 2021-02-11T06:52:01.439 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:52:01.439 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:52:01.439 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:52:01.478 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:52:01.478 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:52:01.709 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=0d061ff0-ede8-4baa-beef-d975d317d505 2021-02-11T06:52:01.751 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613026321.428617_427.1","last_update":"2021-02-11T06:52:01.428617Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:52:01.751 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T06:52:01.751 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:52:01.751 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:52:01.751 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:52:01.790 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:52:01.791 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:52:02.077 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026321.428617_427.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:52:01.428617Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:52:02.078 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T06:52:02.078 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T06:52:02.078 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:52:02.078 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:52:02.078 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:52:02.120 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:52:02.120 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:52:02.403 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026321.428617_427.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:52:01.428617Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:52:02.403 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T06:52:02.403 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:52:02.404 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:52:02.404 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:52:02.404 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:52:02.445 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:52:02.445 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:52:02.710 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026321.428617_427.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:52:01.428617Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:52:02.711 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:52:02.711 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:52:02.711 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:52:02.712 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:52:02.754 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:52:02.754 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:52:03.002 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=0d061ff0-ede8-4baa-beef-d975d317d505 2021-02-11T06:52:03.046 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613026321.428617_427.1","last_update":"2021-02-11T06:52:01.428617Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:52:03.046 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:52:03.046 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:52:03.046 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:52:03.047 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:52:03.088 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:52:03.088 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:52:03.342 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026321.428617_427.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:52:01.428617Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:52:03.343 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:52:03.420 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:52:03.420 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:52:03.421 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:52:03.457 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:52:03.457 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:52:03.700 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=0d061ff0-ede8-4baa-beef-d975d317d505 2021-02-11T06:52:03.739 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613026323.418656_433.1","last_update":"2021-02-11T06:52:03.418656Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:52:03.739 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:52:03.739 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:52:03.739 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:52:03.740 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:52:03.782 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:52:03.782 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:52:04.061 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026323.418656_433.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:52:03.418656Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:52:04.061 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:52:04.207 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-88', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:52:04.207 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-88', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:52:04.207 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-88 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:52:04.263 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:52:04.263 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:52:04.465 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:52:04.461+0000 7f30dafa1080 1 bucket sync caught up with source: 2021-02-11T06:52:04.465 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , , , , , , 00000000005.123.11] 2021-02-11T06:52:04.466 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , , , , , , 00000000005.123.11] 2021-02-11T06:52:04.466 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:52:04.461+0000 7f30dafa1080 0 bucket checkpoint complete 2021-02-11T06:52:04.576 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-88', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:52:04.576 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-88', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:52:04.577 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-88 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:52:04.633 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:52:04.633 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:52:04.866 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:52:04.861+0000 7f04fa72d080 1 bucket sync caught up with source: 2021-02-11T06:52:04.866 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , , , , , , 00000000010.133.5] 2021-02-11T06:52:04.866 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , , , , , , 00000000010.133.5] 2021-02-11T06:52:04.866 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:52:04.861+0000 7f04fa72d080 0 bucket checkpoint complete 2021-02-11T06:52:04.903 DEBUG:tasks.rgw_multi.tests_ps:Event key: "foo" type: "OBJECT_DELETE" 2021-02-11T06:52:04.903 DEBUG:tasks.rgw_multi.tests_ps:Event key: "foo" type: "OBJECT_DELETE" 2021-02-11T06:52:04.903 DEBUG:tasks.rgw_multi.tests_ps:Event key: "foo" type: "OBJECT_DELETE" 2021-02-11T06:52:04.910 DEBUG:tasks.rgw_multi.tests_ps:Event key: "foo" type: "DELETE_MARKER_CREATE" 2021-02-11T06:52:04.910 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-88', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:52:04.911 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-88', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:52:04.911 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-88 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:52:04.953 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:52:04.954 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:52:05.235 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:52:05.229+0000 7f2fef0e7080 1 bucket sync caught up with source: 2021-02-11T06:52:05.236 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , , , , , , 00000000010.133.5] 2021-02-11T06:52:05.236 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , , , , , , 00000000010.133.5] 2021-02-11T06:52:05.236 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:52:05.229+0000 7f2fef0e7080 0 bucket checkpoint complete 2021-02-11T06:52:05.275 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-88', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:52:05.275 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-88', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:52:05.275 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-88 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:52:05.322 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:52:05.322 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:52:05.581 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:52:05.580+0000 7f62c436e080 1 bucket sync caught up with source: 2021-02-11T06:52:27.088 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, , , , , , , , , , 00000000010.133.5] 2021-02-11T06:52:27.088 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, , , , , , , , , , 00000000010.133.5] 2021-02-11T06:52:27.088 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:52:05.580+0000 7f62c436e080 0 bucket checkpoint complete 2021-02-11T06:52:27.090 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-88', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:52:27.090 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-88', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:52:27.091 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-88 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:52:27.139 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:52:27.139 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:52:27.382 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:52:27.376+0000 7f3196624080 1 bucket sync caught up with source: 2021-02-11T06:52:27.382 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , , , , , , 00000000010.133.5] 2021-02-11T06:52:27.382 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , , , , , , 00000000010.133.5] 2021-02-11T06:52:27.382 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:52:27.376+0000 7f3196624080 0 bucket checkpoint complete 2021-02-11T06:52:27.417 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-88', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:52:27.417 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-88', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:52:27.418 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-88 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:52:27.460 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:52:27.461 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:52:27.722 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:52:27.719+0000 7fcde4693080 1 bucket sync caught up with source: 2021-02-11T06:52:27.722 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, , , , , , , , , , 00000000008.115.5] 2021-02-11T06:52:27.722 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, , , , , , , , , , 00000000008.115.5] 2021-02-11T06:52:27.722 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:52:27.719+0000 7fcde4693080 0 bucket checkpoint complete 2021-02-11T06:52:27.757 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-88', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:52:27.757 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-88', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:52:27.757 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-88 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:52:27.850 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:52:27.850 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:52:28.137 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:52:28.131+0000 7fbbe7c68080 1 bucket sync caught up with source: 2021-02-11T06:52:28.137 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, , , , , , , , , , 00000000008.115.5] 2021-02-11T06:52:28.137 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, , , , , , , , , , 00000000008.115.5] 2021-02-11T06:52:28.137 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:52:28.131+0000 7fbbe7c68080 0 bucket checkpoint complete 2021-02-11T06:52:28.169 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-88', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:52:28.169 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-88', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:52:28.169 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xscaju-88 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T06:52:28.215 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:52:28.215 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:52:28.490 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:52:28.487+0000 7f6b29f44080 1 bucket sync caught up with source: 2021-02-11T06:52:28.490 INFO:teuthology.orchestra.run.smithi061.stderr: local status: [, , , , , , , , , , 00000000008.114.5] 2021-02-11T06:52:28.490 INFO:teuthology.orchestra.run.smithi061.stderr: remote markers: [, , , , , , , , , , 00000000008.114.5] 2021-02-11T06:52:28.490 INFO:teuthology.orchestra.run.smithi061.stderr:2021-02-11T06:52:28.487+0000 7f6b29f44080 0 bucket checkpoint complete 2021-02-11T06:52:28.521 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-88', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:52:28.521 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-88', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:52:28.522 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-88 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:52:28.569 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:52:28.569 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:52:28.816 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:52:28.812+0000 7fa128216080 1 bucket sync caught up with source: 2021-02-11T06:52:28.816 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, , , , , , , , , , 00000000008.114.5] 2021-02-11T06:52:28.816 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, , , , , , , , , , 00000000008.114.5] 2021-02-11T06:52:28.816 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:52:28.812+0000 7fa128216080 0 bucket checkpoint complete 2021-02-11T06:52:28.852 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-88 zones={test-zone2, test-zone1} 2021-02-11T06:52:28.869 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:52:28.919 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:52:28.995 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-88 zones={test-zone2, test-zone1} 2021-02-11T06:52:32.477 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-88 zones={test-zone3, test-zone1} 2021-02-11T06:52:32.500 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:52:32.551 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:52:32.620 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-88 zones={test-zone3, test-zone1} 2021-02-11T06:52:32.620 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xscaju-88 zones={test-zone3, test-zone2} 2021-02-11T06:52:32.630 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T06:52:32.679 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T06:52:32.745 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xscaju-88 zones={test-zone3, test-zone2} 2021-02-11T06:52:32.926 INFO:tasks.rgw_multisite_tests:test notification of deletion markers ... ok 2021-02-11T06:52:32.927 INFO:tasks.rgw_multisite_tests:test s3 notification of metadata on master ... ok 2021-02-11T06:52:32.927 INFO:tasks.rgw_multisite_tests:test s3 notification of tags on master ... ok 2021-02-11T06:52:32.927 INFO:tasks.rgw_multisite_tests:test s3 notification of object versions ... ok 2021-02-11T06:52:32.927 INFO:tasks.rgw_multisite_tests:test s3 notification of deletion markers on master ... ok 2021-02-11T06:52:32.928 INFO:tasks.rgw_multisite_tests:test pushing to http endpoint ... ok 2021-02-11T06:52:32.928 INFO:tasks.rgw_multisite_tests:test pushing to http endpoint s3 record format ... ok 2021-02-11T06:52:32.928 INFO:tasks.rgw_multisite_tests:test pushing to amqp endpoint ... ok 2021-02-11T06:52:32.928 INFO:tasks.rgw_multisite_tests:test pushing to amqp endpoint s3 record format ... ok 2021-02-11T06:52:32.928 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:52:32.929 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:52:32.929 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:52:32.969 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:52:32.969 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:52:33.215 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=0d061ff0-ede8-4baa-beef-d975d317d505 2021-02-11T06:52:33.258 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613026352.853505_435.1","last_update":"2021-02-11T06:52:32.853505Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:52:33.258 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T06:52:33.259 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:52:33.259 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:52:33.259 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:52:33.300 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:52:33.300 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:52:33.588 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026352.853505_435.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:52:32.853505Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:52:33.588 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T06:52:33.588 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T06:52:33.589 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:52:33.589 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:52:33.589 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:52:33.635 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:52:33.635 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:52:33.914 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026352.853505_435.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:52:32.853505Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:52:33.914 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T06:52:33.914 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:52:33.914 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:52:33.914 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:52:33.915 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:52:33.957 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:52:33.957 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:52:34.241 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026352.853505_435.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:52:32.853505Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:52:34.242 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:52:34.242 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:52:34.242 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:52:34.243 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:52:34.281 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:52:34.281 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:52:34.544 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=0d061ff0-ede8-4baa-beef-d975d317d505 2021-02-11T06:52:34.581 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613026352.853505_435.1","last_update":"2021-02-11T06:52:32.853505Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:52:34.581 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:52:34.582 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:52:34.582 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:52:34.582 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:52:34.623 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:52:34.623 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:52:34.892 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026352.853505_435.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:52:32.853505Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:52:34.893 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:52:34.948 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:52:34.948 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:52:34.949 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:52:34.986 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:52:34.987 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:52:35.218 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=0d061ff0-ede8-4baa-beef-d975d317d505 2021-02-11T06:52:35.257 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613026354.942181_439.1","last_update":"2021-02-11T06:52:34.942181Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:52:35.258 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:52:35.258 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:52:35.258 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:52:35.259 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:52:35.293 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:52:35.293 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:52:35.557 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026354.942181_439.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:52:34.942181Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:52:35.558 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:52:36.035 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-89', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:52:36.035 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-89', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:52:36.035 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-89 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:52:36.088 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:52:36.088 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:52:36.332 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:52:36.324+0000 7f4b06bc7080 1 bucket sync caught up with source: 2021-02-11T06:52:36.332 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000002.94.5, , 00000000001.90.5, 00000000001.138.5, 00000000001.128.5, , 00000000001.136.5, 00000000001.127.5, 00000000001.89.5, 00000000002.99.5] 2021-02-11T06:52:36.332 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000002.94.5, , 00000000001.90.5, 00000000001.138.5, 00000000001.128.5, , 00000000001.136.5, 00000000001.127.5, 00000000001.89.5, 00000000002.99.5] 2021-02-11T06:52:36.332 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:52:36.324+0000 7f4b06bc7080 0 bucket checkpoint complete 2021-02-11T06:52:36.482 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-89', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:52:36.483 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'xscaju-89', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T06:52:36.483 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket xscaju-89 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T06:52:36.523 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:52:36.523 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:52:36.886 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:52:36.880+0000 7fd514582080 1 bucket sync caught up with source: 2021-02-11T06:52:36.966 INFO:teuthology.orchestra.run.smithi172.stderr: local status: [, 00000000004.100.5, , 00000000002.96.5, 00000000002.140.5, 00000000002.130.5, , 00000000002.138.5, 00000000002.129.5, 00000000002.91.5, 00000000004.103.5] 2021-02-11T06:52:36.966 INFO:teuthology.orchestra.run.smithi172.stderr: remote markers: [, 00000000004.100.5, , 00000000002.96.5, 00000000002.140.5, 00000000002.130.5, , 00000000002.138.5, 00000000002.129.5, 00000000002.91.5, 00000000004.103.5] 2021-02-11T06:52:36.966 INFO:teuthology.orchestra.run.smithi172.stderr:2021-02-11T06:52:36.880+0000 7fd514582080 0 bucket checkpoint complete 2021-02-11T06:52:37.073 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:52:37.073 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:52:37.073 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:52:37.108 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:52:37.109 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:52:37.375 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=0d061ff0-ede8-4baa-beef-d975d317d505 2021-02-11T06:52:37.403 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613026357.060498_530.1","last_update":"2021-02-11T06:52:37.060498Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:52:37.404 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:52:37.404 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:52:37.404 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:52:37.404 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:52:37.447 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:52:37.448 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:52:37.711 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026357.060498_530.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:52:37.060498Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:52:37.712 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:52:37.720 WARNING:tasks.rgw_multi.tests_ps:superfluous records are found 2021-02-11T06:52:37.746 INFO:tasks.rgw_multisite_tests:test notification status upon bucket deletion ... ok 2021-02-11T06:52:37.747 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:52:37.747 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:52:37.747 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:52:37.787 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:52:37.787 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:52:38.018 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=0d061ff0-ede8-4baa-beef-d975d317d505 2021-02-11T06:52:38.054 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613026357.060498_530.1","last_update":"2021-02-11T06:52:37.060498Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:52:38.054 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T06:52:38.054 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:52:38.054 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:52:38.055 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:52:38.097 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:52:38.097 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:52:38.362 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026357.060498_530.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:52:37.060498Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:52:38.363 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T06:52:38.363 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T06:52:38.363 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:52:38.363 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:52:38.364 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:52:38.406 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:52:38.406 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:52:38.693 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026357.060498_530.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:52:37.060498Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:52:38.693 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T06:52:38.693 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:52:38.694 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:52:38.694 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:52:38.694 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:52:38.734 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:52:38.734 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:52:39.026 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026357.060498_530.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:52:37.060498Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:52:39.026 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:52:39.027 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:52:39.027 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:52:39.027 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:52:39.067 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:52:59.550 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:52:59.551 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=0d061ff0-ede8-4baa-beef-d975d317d505 2021-02-11T06:52:59.551 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613026357.060498_530.1","last_update":"2021-02-11T06:52:37.060498Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:52:59.552 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:52:59.552 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:52:59.552 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:52:59.552 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:52:59.593 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:52:59.593 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:52:59.864 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026357.060498_530.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:52:37.060498Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:52:59.864 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:52:59.915 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:52:59.915 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:52:59.916 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:52:59.952 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:52:59.952 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:53:00.180 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=0d061ff0-ede8-4baa-beef-d975d317d505 2021-02-11T06:53:06.645 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1613026379.910770_535.1","last_update":"2021-02-11T06:52:59.910770Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T06:53:06.646 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T06:53:06.646 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:53:06.646 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T06:53:06.646 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T06:53:06.685 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2021-02-11T06:53:06.686 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T06:53:06.954 INFO:teuthology.orchestra.run.smithi172.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d061ff0-ede8-4baa-beef-d975d317d505","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613026379.910770_535.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T06:52:59.910770Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613024301.142235_74.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T06:18:21.142235Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T06:53:06.954 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T06:53:06.965 INFO:tasks.rgw_multi.tests_ps:missing topic is expected 2021-02-11T06:53:07.054 INFO:tasks.rgw_multisite_tests:test creating a subscription when no topic info exists ... ok 2021-02-11T06:53:07.054 INFO:tasks.rgw_multisite_tests:test updating topic associated with a notification ... ok 2021-02-11T06:53:07.055 INFO:tasks.rgw_multisite_tests:test updating the topic of a notification ... ok 2021-02-11T06:53:07.055 INFO:tasks.rgw_multisite_tests:test notification creation with multiple topics ... ok 2021-02-11T06:53:07.055 INFO:tasks.rgw_multisite_tests: 2021-02-11T06:53:07.056 INFO:tasks.rgw_multisite_tests:====================================================================== 2021-02-11T06:53:07.056 INFO:tasks.rgw_multisite_tests:FAIL: test radosgw-admin commands 2021-02-11T06:53:07.056 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2021-02-11T06:53:07.056 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2021-02-11T06:53:07.056 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2021-02-11T06:53:07.056 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2021-02-11T06:53:07.056 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_84a8025e5a892a6879f75e17e03444bf443eae4b/qa/tasks/rgw_multi/tests_ps.py", line 3062, in test_ps_admin 2021-02-11T06:53:07.057 INFO:tasks.rgw_multisite_tests: assert_equal(len(parsed_result['events']), 10) 2021-02-11T06:53:07.057 INFO:tasks.rgw_multisite_tests:AssertionError: 11 != 10 2021-02-11T06:53:07.057 INFO:tasks.rgw_multisite_tests: 2021-02-11T06:53:07.057 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2021-02-11T06:53:07.057 INFO:tasks.rgw_multisite_tests:Ran 55 tests in 296.385s 2021-02-11T06:53:07.057 INFO:tasks.rgw_multisite_tests: 2021-02-11T06:53:07.057 INFO:tasks.rgw_multisite_tests:FAILED (failures=1) 2021-02-11T06:53:07.058 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/run_tasks.py", line 94, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_ceph-c_84a8025e5a892a6879f75e17e03444bf443eae4b/qa/tasks/rgw_multisite_tests.py", line 74, in begin raise RuntimeError(error_msg + 'test failures') RuntimeError: rgw multisite pubsub, test failures 2021-02-11T06:53:07.200 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=35a2c565587343489a6c9ca0d127d868 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/run_tasks.py", line 94, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_ceph-c_84a8025e5a892a6879f75e17e03444bf443eae4b/qa/tasks/rgw_multisite_tests.py", line 74, in begin raise RuntimeError(error_msg + 'test failures') RuntimeError: rgw multisite pubsub, test failures 2021-02-11T06:53:07.201 DEBUG:teuthology.run_tasks:Unwinding manager rgw-multisite-tests 2021-02-11T06:53:07.236 DEBUG:teuthology.run_tasks:Unwinding manager rgw-multisite 2021-02-11T06:53:07.608 DEBUG:teuthology.run_tasks:Unwinding manager rgw 2021-02-11T06:53:07.703 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2021-02-11T06:53:07.703 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T06:53:07.723 INFO:tasks.rgw.c1.client.0.smithi061.stdout:2021-02-11T06:53:07.699+0000 7f1760a62700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 18780) UID: 0 2021-02-11T06:53:07.723 INFO:tasks.rgw.c1.client.0.smithi061.stdout:2021-02-11T06:53:07.699+0000 7f17785890c0 -1 shutting down 2021-02-11T06:53:08.252 INFO:tasks.rgw.c1.client.1.smithi061.stdout:2021-02-11T06:53:08.247+0000 7f6f817fa700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-5) 2021-02-11T06:53:08.300 INFO:tasks.rgw.c2.client.0.smithi172.stdout:2021-02-11T06:53:08.295+0000 7fb3137fe700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-5) 2021-02-11T06:53:13.530 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2021-02-11T06:53:13.804 INFO:tasks.rgw.c1.client.0:Stopped 2021-02-11T06:53:39.265 DEBUG:teuthology.orchestra.run.smithi061:> rm -f /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock 2021-02-11T06:53:39.271 DEBUG:teuthology.orchestra.run.smithi061:> rm -f /home/ubuntu/cephtest/vault-token 2021-02-11T06:53:39.318 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2021-02-11T06:53:39.319 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T06:53:39.362 INFO:tasks.rgw.c1.client.1.smithi061.stdout:2021-02-11T06:53:39.358+0000 7f7043fff700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 32050) UID: 0 2021-02-11T06:53:39.362 INFO:tasks.rgw.c1.client.1.smithi061.stdout:2021-02-11T06:53:39.358+0000 7f705fbd40c0 -1 shutting down 2021-02-11T06:53:39.770 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2021-02-11T06:53:39.770 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2021-02-11T06:53:45.277 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~6s 2021-02-11T06:53:45.921 INFO:tasks.rgw.c1.client.1:Stopped 2021-02-11T06:53:45.922 DEBUG:teuthology.orchestra.run.smithi061:> rm -f /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock 2021-02-11T06:53:45.929 DEBUG:teuthology.orchestra.run.smithi061:> rm -f /home/ubuntu/cephtest/vault-token 2021-02-11T06:53:45.978 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2021-02-11T06:53:45.979 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T06:53:45.980 INFO:tasks.rgw.c2.client.0.smithi172.stdout:2021-02-11T06:53:45.978+0000 7fb3ee1d0700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 18730) UID: 0 2021-02-11T06:53:45.980 INFO:tasks.rgw.c2.client.0.smithi172.stdout:2021-02-11T06:53:45.978+0000 7fb405cf70c0 -1 shutting down 2021-02-11T06:53:51.423 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2021-02-11T06:53:51.932 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2021-02-11T06:53:52.080 INFO:tasks.rgw.c2.client.0:Stopped 2021-02-11T06:53:52.080 DEBUG:teuthology.orchestra.run.smithi172:> rm -f /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock 2021-02-11T06:53:52.087 DEBUG:teuthology.orchestra.run.smithi172:> rm -f /home/ubuntu/cephtest/vault-token 2021-02-11T06:53:52.133 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2021-02-11T06:53:52.133 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T06:53:52.176 INFO:tasks.rgw.c2.client.1.smithi172.stdout:2021-02-11T06:53:52.174+0000 7f2becee4700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 19887) UID: 0 2021-02-11T06:53:52.177 INFO:tasks.rgw.c2.client.1.smithi172.stdout:2021-02-11T06:53:52.174+0000 7f2c04a0b0c0 -1 shutting down 2021-02-11T06:53:57.435 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2021-02-11T06:54:14.615 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2021-02-11T06:54:14.616 INFO:tasks.rgw.c2.client.1:Stopped 2021-02-11T06:54:14.617 DEBUG:teuthology.orchestra.run.smithi172:> rm -f /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock 2021-02-11T06:54:14.624 DEBUG:teuthology.orchestra.run.smithi172:> rm -f /home/ubuntu/cephtest/vault-token 2021-02-11T06:54:14.673 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2021-02-11T06:54:14.683 INFO:tasks.ceph.ceph_manager.c2:Canceling any pending splits or merges... 2021-02-11T06:54:14.683 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2021-02-11T06:54:15.023 INFO:teuthology.orchestra.run.smithi172.stdout: 2021-02-11T06:54:15.024 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":73,"fsid":"0d6e87fb-daa0-4c85-a370-bf71998465da","created":"2021-02-11T06:11:49.134481+0000","modified":"2021-02-11T06:34:48.270307+0000","last_up_change":"2021-02-11T06:11:56.924613+0000","last_in_change":"2021-02-11T06:11:56.924613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-02-11T06:11:56.930707+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-02-11T06:12:04.328307+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"default.rgw.buckets.data","create_time":"2021-02-11T06:12:39.417876+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":4,"pool_name":"default.rgw.buckets.index","create_time":"2021-02-11T06:12:41.488780+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":".rgw.root","create_time":"2021-02-11T06:13:37.173346+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.log","create_time":"2021-02-11T06:13:39.262940+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":7,"pool_name":"default.rgw.control","create_time":"2021-02-11T06:13:41.226614+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":8,"pool_name":"default.rgw.meta","create_time":"2021-02-11T06:13:43.305908+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":8},"application_metadata":{"rgw":{}}},{"pool":9,"pool_name":"test-zone2.rgw.buckets.data","create_time":"2021-02-11T06:14:16.874230+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":10,"pool_name":"test-zone2.rgw.log","create_time":"2021-02-11T06:14:18.430298+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":11,"pool_name":"test-zone2.rgw.control","create_time":"2021-02-11T06:14:20.242633+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":12,"pool_name":"test-zone2.rgw.meta","create_time":"2021-02-11T06:14:22.316980+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":8},"application_metadata":{"rgw":{}}},{"pool":13,"pool_name":"test-zone4.rgw.buckets.data","create_time":"2021-02-11T06:14:58.328467+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":14,"pool_name":"test-zone4.rgw.log","create_time":"2021-02-11T06:15:15.468045+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":15,"pool_name":"test-zone4.rgw.control","create_time":"2021-02-11T06:15:17.269593+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":16,"pool_name":"test-zone4.rgw.meta","create_time":"2021-02-11T06:15:19.341086+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":8},"application_metadata":{"rgw":{}}},{"pool":17,"pool_name":"test-zone2.rgw.buckets.index","create_time":"2021-02-11T06:17:42.135088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":8},"application_metadata":{"rgw":{}}},{"pool":18,"pool_name":"test-zone4.rgw.buckets.index","create_time":"2021-02-11T06:17:42.208996+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":8},"application_metadata":{"rgw":{}}},{"pool":19,"pool_name":"test-zone2.rgw.otp","create_time":"2021-02-11T06:25:58.985045+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":20,"pool_name":"remove.rgw.log","create_time":"2021-02-11T06:34:40.907852+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":21,"pool_name":"remove.rgw.control","create_time":"2021-02-11T06:34:42.291205+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":22,"pool_name":"remove.rgw.meta","create_time":"2021-02-11T06:34:44.354422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":8},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"c5cdebd7-634e-47c8-98cb-ab4dfea1e594","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":14675},{"type":"v1","addr":"172.21.15.172:6801","nonce":14675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":14675},{"type":"v1","addr":"172.21.15.172:6803","nonce":14675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":14675},{"type":"v1","addr":"172.21.15.172:6807","nonce":14675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":14675},{"type":"v1","addr":"172.21.15.172:6805","nonce":14675}]},"public_addr":"172.21.15.172:6801/14675","cluster_addr":"172.21.15.172:6803/14675","heartbeat_back_addr":"172.21.15.172:6807/14675","heartbeat_front_addr":"172.21.15.172:6805/14675","state":["exists","up"]},{"osd":1,"uuid":"85ae3f4d-8dd9-4363-8a1f-d31d71884fe8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":14677},{"type":"v1","addr":"172.21.15.172:6809","nonce":14677}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":14677},{"type":"v1","addr":"172.21.15.172:6812","nonce":14677}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":14677},{"type":"v1","addr":"172.21.15.172:6819","nonce":14677}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":14677},{"type":"v1","addr":"172.21.15.172:6815","nonce":14677}]},"public_addr":"172.21.15.172:6809/14677","cluster_addr":"172.21.15.172:6812/14677","heartbeat_back_addr":"172.21.15.172:6819/14677","heartbeat_front_addr":"172.21.15.172:6815/14677","state":["exists","up"]},{"osd":2,"uuid":"3a60e445-b4ef-42ea-8328-496da35e31b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6811","nonce":14676},{"type":"v1","addr":"172.21.15.172:6813","nonce":14676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":14676},{"type":"v1","addr":"172.21.15.172:6818","nonce":14676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":14676},{"type":"v1","addr":"172.21.15.172:6823","nonce":14676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":14676},{"type":"v1","addr":"172.21.15.172:6821","nonce":14676}]},"public_addr":"172.21.15.172:6813/14676","cluster_addr":"172.21.15.172:6818/14676","heartbeat_back_addr":"172.21.15.172:6823/14676","heartbeat_front_addr":"172.21.15.172:6821/14676","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T06:11:54.944094+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T06:11:54.812675+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T06:11:54.941007+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2b","mappings":[{"from":0,"to":1}]},{"pgid":"3.34","mappings":[{"from":0,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.6","mappings":[{"from":1,"to":2}]},{"pgid":"4.9","mappings":[{"from":1,"to":2}]},{"pgid":"4.20","mappings":[{"from":1,"to":0}]},{"pgid":"4.32","mappings":[{"from":1,"to":0}]},{"pgid":"4.3c","mappings":[{"from":1,"to":2}]},{"pgid":"4.3e","mappings":[{"from":1,"to":0}]},{"pgid":"10.18","mappings":[{"from":2,"to":0}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-02-11T06:54:15.039 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster c2 config set global mon_health_to_clog false 2021-02-11T06:54:15.381 INFO:teuthology.misc:Shutting down mds daemons... 2021-02-11T06:54:15.381 INFO:teuthology.misc:Shutting down osd daemons... 2021-02-11T06:54:15.382 DEBUG:tasks.ceph.c2.osd.0:waiting for process to exit 2021-02-11T06:54:15.382 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T06:54:15.454 INFO:tasks.ceph.c2.osd.0:Stopped 2021-02-11T06:54:15.455 DEBUG:tasks.ceph.c2.osd.1:waiting for process to exit 2021-02-11T06:54:15.455 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T06:54:15.533 INFO:tasks.ceph.c2.osd.1:Stopped 2021-02-11T06:54:15.533 DEBUG:tasks.ceph.c2.osd.2:waiting for process to exit 2021-02-11T06:54:15.533 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T06:54:15.622 INFO:tasks.ceph.c2.osd.2:Stopped 2021-02-11T06:54:15.622 INFO:teuthology.misc:Shutting down mgr daemons... 2021-02-11T06:54:15.622 DEBUG:tasks.ceph.c2.mgr.x:waiting for process to exit 2021-02-11T06:54:15.623 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T06:54:15.647 INFO:tasks.ceph.c2.mgr.x:Stopped 2021-02-11T06:54:15.648 INFO:teuthology.misc:Shutting down mon daemons... 2021-02-11T06:54:15.648 DEBUG:tasks.ceph.c2.mon.a:waiting for process to exit 2021-02-11T06:54:15.648 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T06:54:15.719 INFO:tasks.ceph.c2.mon.a:Stopped 2021-02-11T06:54:15.719 INFO:tasks.ceph:Checking cluster log for badness... 2021-02-11T06:54:15.720 DEBUG:teuthology.orchestra.run.smithi172:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | head -n 1 2021-02-11T06:54:15.777 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-0 on ubuntu@smithi172.front.sepia.ceph.com 2021-02-11T06:54:15.778 DEBUG:teuthology.orchestra.run.smithi172:> sync && sudo umount -f /var/lib/ceph/osd/c2-0 2021-02-11T06:54:16.591 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-1 on ubuntu@smithi172.front.sepia.ceph.com 2021-02-11T06:54:16.592 DEBUG:teuthology.orchestra.run.smithi172:> sync && sudo umount -f /var/lib/ceph/osd/c2-1 2021-02-11T06:54:16.798 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-2 on ubuntu@smithi172.front.sepia.ceph.com 2021-02-11T06:54:16.799 DEBUG:teuthology.orchestra.run.smithi172:> sync && sudo umount -f /var/lib/ceph/osd/c2-2 2021-02-11T06:54:16.934 INFO:tasks.ceph:Archiving mon data... 2021-02-11T06:54:16.936 DEBUG:teuthology.misc:Transferring archived files from smithi172:/var/lib/ceph/mon/c2-a to /home/teuthworker/archive/aemerson-2021-02-11_05:45:06-rgw-wip-easter-bunny-distro-basic-smithi/5874443/data/c2.mon.a.tgz 2021-02-11T06:54:16.936 DEBUG:teuthology.orchestra.run.smithi172:> mktemp 2021-02-11T06:54:16.941 INFO:teuthology.orchestra.run.smithi172.stdout:/tmp/tmp.huAyqr7Lel 2021-02-11T06:54:16.942 DEBUG:teuthology.orchestra.run.smithi172:> sudo tar cz -f - -C /var/lib/ceph/mon/c2-a -- . > /tmp/tmp.huAyqr7Lel 2021-02-11T06:54:17.335 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 0666 /tmp/tmp.huAyqr7Lel 2021-02-11T06:54:17.400 DEBUG:teuthology.orchestra.remote:smithi172:/tmp/tmp.huAyqr7Lel is 2MB 2021-02-11T06:54:17.483 DEBUG:teuthology.orchestra.run.smithi172:> rm -fr /tmp/tmp.huAyqr7Lel 2021-02-11T06:54:17.490 INFO:tasks.ceph:Cleaning ceph cluster... 2021-02-11T06:54:17.490 DEBUG:teuthology.orchestra.run.smithi061:> sudo rm -rf -- /etc/ceph/c2.conf /etc/ceph/c2.keyring /home/ubuntu/cephtest/c2.data /home/ubuntu/cephtest/c2.monmap /home/ubuntu/cephtest/../*.pid 2021-02-11T06:54:17.492 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -rf -- /etc/ceph/c2.conf /etc/ceph/c2.keyring /home/ubuntu/cephtest/c2.data /home/ubuntu/cephtest/c2.monmap /home/ubuntu/cephtest/../*.pid 2021-02-11T06:54:17.546 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2021-02-11T06:54:17.708 INFO:tasks.ceph.ceph_manager.c1:Canceling any pending splits or merges... 2021-02-11T06:54:17.708 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2021-02-11T06:54:18.016 INFO:teuthology.orchestra.run.smithi061.stdout: 2021-02-11T06:54:18.016 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":70,"fsid":"043de53e-a645-454b-bce7-c489302846ce","created":"2021-02-11T06:10:51.924984+0000","modified":"2021-02-11T06:46:14.932559+0000","last_up_change":"2021-02-11T06:10:59.194103+0000","last_in_change":"2021-02-11T06:10:59.194103+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-02-11T06:10:59.208467+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-02-11T06:11:07.282762+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"default.rgw.buckets.data","create_time":"2021-02-11T06:12:17.080042+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":4,"pool_name":"default.rgw.buckets.index","create_time":"2021-02-11T06:12:34.389345+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":".rgw.root","create_time":"2021-02-11T06:13:05.871859+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.log","create_time":"2021-02-11T06:13:07.141654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":7,"pool_name":"default.rgw.control","create_time":"2021-02-11T06:13:09.105374+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":8,"pool_name":"default.rgw.meta","create_time":"2021-02-11T06:13:11.205539+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"36","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":8},"application_metadata":{"rgw":{}}},{"pool":9,"pool_name":"test-zone1.rgw.buckets.data","create_time":"2021-02-11T06:13:56.197757+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":10,"pool_name":"test-zone1.rgw.log","create_time":"2021-02-11T06:13:57.481174+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":11,"pool_name":"test-zone1.rgw.control","create_time":"2021-02-11T06:13:59.322285+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":12,"pool_name":"test-zone1.rgw.meta","create_time":"2021-02-11T06:14:01.411146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":8},"application_metadata":{"rgw":{}}},{"pool":13,"pool_name":"test-zone1.rgw.otp","create_time":"2021-02-11T06:14:35.285966+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":14,"pool_name":"test-zone3.rgw.buckets.data","create_time":"2021-02-11T06:14:36.131656+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":15,"pool_name":"test-zone3.rgw.log","create_time":"2021-02-11T06:14:38.366206+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":16,"pool_name":"test-zone3.rgw.control","create_time":"2021-02-11T06:14:40.170537+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":17,"pool_name":"test-zone3.rgw.meta","create_time":"2021-02-11T06:14:42.248767+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":8},"application_metadata":{"rgw":{}}},{"pool":18,"pool_name":"test-zone1.rgw.buckets.index","create_time":"2021-02-11T06:17:38.331662+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":8},"application_metadata":{"rgw":{}}},{"pool":19,"pool_name":"test-zone3.rgw.buckets.index","create_time":"2021-02-11T06:17:42.201937+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":8},"application_metadata":{"rgw":{}}},{"pool":20,"pool_name":"test-zone1.rgw.buckets.non-ec","create_time":"2021-02-11T06:46:12.345616+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"0d97d5d6-5f78-4266-9bb1-8f868bd62e54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":14659},{"type":"v1","addr":"172.21.15.61:6809","nonce":14659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":14659},{"type":"v1","addr":"172.21.15.61:6811","nonce":14659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":14659},{"type":"v1","addr":"172.21.15.61:6815","nonce":14659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":14659},{"type":"v1","addr":"172.21.15.61:6813","nonce":14659}]},"public_addr":"172.21.15.61:6809/14659","cluster_addr":"172.21.15.61:6811/14659","heartbeat_back_addr":"172.21.15.61:6815/14659","heartbeat_front_addr":"172.21.15.61:6813/14659","state":["exists","up"]},{"osd":1,"uuid":"2b921cd0-ae04-4e00-abee-e6327aaff01f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":14661},{"type":"v1","addr":"172.21.15.61:6801","nonce":14661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":14661},{"type":"v1","addr":"172.21.15.61:6803","nonce":14661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":14661},{"type":"v1","addr":"172.21.15.61:6807","nonce":14661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":14661},{"type":"v1","addr":"172.21.15.61:6805","nonce":14661}]},"public_addr":"172.21.15.61:6801/14661","cluster_addr":"172.21.15.61:6803/14661","heartbeat_back_addr":"172.21.15.61:6807/14661","heartbeat_front_addr":"172.21.15.61:6805/14661","state":["exists","up"]},{"osd":2,"uuid":"1d1b38e0-b970-4074-9525-26fef2f47dad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":14660},{"type":"v1","addr":"172.21.15.61:6817","nonce":14660}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":14660},{"type":"v1","addr":"172.21.15.61:6819","nonce":14660}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":14660},{"type":"v1","addr":"172.21.15.61:6823","nonce":14660}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":14660},{"type":"v1","addr":"172.21.15.61:6821","nonce":14660}]},"public_addr":"172.21.15.61:6817/14660","cluster_addr":"172.21.15.61:6819/14660","heartbeat_back_addr":"172.21.15.61:6823/14660","heartbeat_front_addr":"172.21.15.61:6821/14660","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T06:10:57.694212+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T06:10:57.074765+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T06:10:57.729633+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.24","mappings":[{"from":0,"to":1}]},{"pgid":"3.2a","mappings":[{"from":0,"to":2}]},{"pgid":"4.f","mappings":[{"from":1,"to":0}]},{"pgid":"4.15","mappings":[{"from":1,"to":2}]},{"pgid":"4.24","mappings":[{"from":1,"to":0}]},{"pgid":"4.30","mappings":[{"from":1,"to":2}]},{"pgid":"4.31","mappings":[{"from":1,"to":2}]},{"pgid":"4.38","mappings":[{"from":1,"to":2}]},{"pgid":"4.3b","mappings":[{"from":1,"to":2}]},{"pgid":"10.19","mappings":[{"from":1,"to":0}]},{"pgid":"14.1a","mappings":[{"from":1,"to":2}]},{"pgid":"14.2d","mappings":[{"from":1,"to":0}]},{"pgid":"14.31","mappings":[{"from":1,"to":2}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-02-11T06:54:18.031 DEBUG:teuthology.orchestra.run.smithi061:> sudo ceph --cluster c1 config set global mon_health_to_clog false 2021-02-11T06:54:18.356 INFO:teuthology.misc:Shutting down mds daemons... 2021-02-11T06:54:18.356 INFO:teuthology.misc:Shutting down osd daemons... 2021-02-11T06:54:18.356 DEBUG:tasks.ceph.c1.osd.0:waiting for process to exit 2021-02-11T06:54:18.357 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T06:54:18.417 INFO:tasks.ceph.c1.osd.0:Stopped 2021-02-11T06:54:18.418 DEBUG:tasks.ceph.c1.osd.1:waiting for process to exit 2021-02-11T06:54:18.418 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T06:54:18.503 INFO:tasks.ceph.c1.osd.1:Stopped 2021-02-11T06:54:18.503 DEBUG:tasks.ceph.c1.osd.2:waiting for process to exit 2021-02-11T06:54:18.504 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T06:54:18.588 INFO:tasks.ceph.c1.osd.2:Stopped 2021-02-11T06:54:18.588 INFO:teuthology.misc:Shutting down mgr daemons... 2021-02-11T06:54:18.589 DEBUG:tasks.ceph.c1.mgr.x:waiting for process to exit 2021-02-11T06:54:18.589 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T06:54:18.620 INFO:tasks.ceph.c1.mgr.x:Stopped 2021-02-11T06:54:18.621 INFO:teuthology.misc:Shutting down mon daemons... 2021-02-11T06:54:18.621 DEBUG:tasks.ceph.c1.mon.a:waiting for process to exit 2021-02-11T06:54:18.621 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T06:54:18.684 INFO:tasks.ceph.c1.mon.a:Stopped 2021-02-11T06:54:18.684 INFO:tasks.ceph:Checking cluster log for badness... 2021-02-11T06:54:18.685 DEBUG:teuthology.orchestra.run.smithi061:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | head -n 1 2021-02-11T06:54:18.744 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-0 on ubuntu@smithi061.front.sepia.ceph.com 2021-02-11T06:54:18.745 DEBUG:teuthology.orchestra.run.smithi061:> sync && sudo umount -f /var/lib/ceph/osd/c1-0 2021-02-11T06:54:19.156 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-1 on ubuntu@smithi061.front.sepia.ceph.com 2021-02-11T06:54:28.383 DEBUG:teuthology.orchestra.run.smithi061:> sync && sudo umount -f /var/lib/ceph/osd/c1-1 2021-02-11T06:54:28.384 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 has been restored 2021-02-11T06:54:28.560 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-2 on ubuntu@smithi061.front.sepia.ceph.com 2021-02-11T06:54:28.560 DEBUG:teuthology.orchestra.run.smithi061:> sync && sudo umount -f /var/lib/ceph/osd/c1-2 2021-02-11T06:54:28.712 INFO:tasks.ceph:Archiving mon data... 2021-02-11T06:54:28.712 DEBUG:teuthology.misc:Transferring archived files from smithi061:/var/lib/ceph/mon/c1-a to /home/teuthworker/archive/aemerson-2021-02-11_05:45:06-rgw-wip-easter-bunny-distro-basic-smithi/5874443/data/c1.mon.a.tgz 2021-02-11T06:54:28.713 DEBUG:teuthology.orchestra.run.smithi061:> mktemp 2021-02-11T06:54:28.719 INFO:teuthology.orchestra.run.smithi061.stdout:/tmp/tmp.V2RkyOjND7 2021-02-11T06:54:28.719 DEBUG:teuthology.orchestra.run.smithi061:> sudo tar cz -f - -C /var/lib/ceph/mon/c1-a -- . > /tmp/tmp.V2RkyOjND7 2021-02-11T06:54:29.129 DEBUG:teuthology.orchestra.run.smithi061:> sudo chmod 0666 /tmp/tmp.V2RkyOjND7 2021-02-11T06:54:29.193 DEBUG:teuthology.orchestra.remote:smithi061:/tmp/tmp.V2RkyOjND7 is 2MB 2021-02-11T06:54:29.273 DEBUG:teuthology.orchestra.run.smithi061:> rm -fr /tmp/tmp.V2RkyOjND7 2021-02-11T06:54:29.280 INFO:tasks.ceph:Cleaning ceph cluster... 2021-02-11T06:54:29.280 DEBUG:teuthology.orchestra.run.smithi061:> sudo rm -rf -- /etc/ceph/c1.conf /etc/ceph/c1.keyring /home/ubuntu/cephtest/c1.data /home/ubuntu/cephtest/c1.monmap /home/ubuntu/cephtest/../*.pid 2021-02-11T06:54:29.322 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -rf -- /etc/ceph/c1.conf /etc/ceph/c1.keyring /home/ubuntu/cephtest/c1.data /home/ubuntu/cephtest/c1.monmap /home/ubuntu/cephtest/../*.pid 2021-02-11T06:54:29.338 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2021-02-11T06:54:29.338 DEBUG:teuthology.orchestra.run.smithi061:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2021-02-11T06:54:29.378 DEBUG:teuthology.orchestra.run.smithi172:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2021-02-11T06:54:29.403 INFO:teuthology.orchestra.run.smithi061.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2021-02-11T06:54:29.403 INFO:teuthology.orchestra.run.smithi172.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2021-02-11T06:54:29.412 INFO:tasks.ceph:Archiving crash dumps... 2021-02-11T06:54:29.414 DEBUG:teuthology.misc:Transferring archived files from smithi061:/var/lib/ceph/crash to /home/teuthworker/archive/aemerson-2021-02-11_05:45:06-rgw-wip-easter-bunny-distro-basic-smithi/5874443/remote/smithi061/crash 2021-02-11T06:54:29.415 DEBUG:teuthology.orchestra.run.smithi061:> sudo tar cz -f - -C /var/lib/ceph/crash -- . 2021-02-11T06:54:29.471 DEBUG:teuthology.misc:Transferring archived files from smithi172:/var/lib/ceph/crash to /home/teuthworker/archive/aemerson-2021-02-11_05:45:06-rgw-wip-easter-bunny-distro-basic-smithi/5874443/remote/smithi172/crash 2021-02-11T06:54:29.472 DEBUG:teuthology.orchestra.run.smithi172:> sudo tar cz -f - -C /var/lib/ceph/crash -- . 2021-02-11T06:54:29.490 INFO:tasks.ceph:Compressing logs... 2021-02-11T06:54:29.491 DEBUG:teuthology.orchestra.run.smithi061:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2021-02-11T06:54:29.514 DEBUG:teuthology.orchestra.run.smithi172:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2021-02-11T06:56:10.660 INFO:tasks.ceph:Archiving logs... 2021-02-11T06:56:10.741 DEBUG:teuthology.misc:Transferring archived files from smithi061:/var/log/ceph to /home/teuthworker/archive/aemerson-2021-02-11_05:45:06-rgw-wip-easter-bunny-distro-basic-smithi/5874443/remote/smithi061/log 2021-02-11T06:56:10.742 DEBUG:teuthology.orchestra.run.smithi061:> sudo tar cz -f - -C /var/log/ceph -- . 2021-02-11T06:56:33.763 DEBUG:teuthology.misc:Transferring archived files from smithi172:/var/log/ceph to /home/teuthworker/archive/aemerson-2021-02-11_05:45:06-rgw-wip-easter-bunny-distro-basic-smithi/5874443/remote/smithi172/log 2021-02-11T06:56:33.764 DEBUG:teuthology.orchestra.run.smithi172:> sudo tar cz -f - -C /var/log/ceph -- . 2021-02-11T06:56:56.784 DEBUG:teuthology.run_tasks:Unwinding manager install 2021-02-11T06:57:09.088 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits... 2021-02-11T06:57:09.089 DEBUG:teuthology.orchestra.run.smithi061:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2021-02-11T06:57:09.093 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2021-02-11T06:57:09.151 DEBUG:teuthology.run_tasks:Unwinding manager clock 2021-02-11T06:57:10.909 INFO:teuthology.task.clock:Checking final clock skew... 2021-02-11T06:57:10.910 DEBUG:teuthology.orchestra.run.smithi061:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-02-11T06:57:10.931 INFO:teuthology.orchestra.run.smithi061.stdout: remote refid st t when poll reach delay offset jitter 2021-02-11T06:57:10.931 INFO:teuthology.orchestra.run.smithi061.stdout:============================================================================== 2021-02-11T06:57:10.932 INFO:teuthology.orchestra.run.smithi061.stdout:-hv01.front.sepi 216.240.36.24 3 u 74 128 377 0.083 1.422 0.359 2021-02-11T06:57:10.932 INFO:teuthology.orchestra.run.smithi061.stdout:*hv02.front.sepi 163.237.218.18 2 u 122 128 377 0.088 -0.287 0.125 2021-02-11T06:57:10.932 INFO:teuthology.orchestra.run.smithi061.stdout:+hv03.front.sepi 74.6.168.72 3 u 58 128 377 0.099 -0.735 0.183 2021-02-11T06:57:10.932 INFO:teuthology.orchestra.run.smithi061.stdout:+hv04.front.sepi 144.172.126.201 3 u 77 128 377 0.069 -0.609 0.226 2021-02-11T06:57:10.933 DEBUG:teuthology.orchestra.run.smithi172:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-02-11T06:57:10.954 INFO:teuthology.orchestra.run.smithi172.stdout: remote refid st t when poll reach delay offset jitter 2021-02-11T06:57:10.955 INFO:teuthology.orchestra.run.smithi172.stdout:============================================================================== 2021-02-11T06:57:10.955 INFO:teuthology.orchestra.run.smithi172.stdout:-hv01.front.sepi 216.240.36.24 3 u 115 128 377 0.102 3.534 0.367 2021-02-11T06:57:10.955 INFO:teuthology.orchestra.run.smithi172.stdout:*hv02.front.sepi 163.237.218.18 2 u 118 128 377 0.075 1.662 0.496 2021-02-11T06:57:10.955 INFO:teuthology.orchestra.run.smithi172.stdout:+hv03.front.sepi 74.6.168.72 3 u 120 128 377 0.080 1.397 0.532 2021-02-11T06:57:10.955 INFO:teuthology.orchestra.run.smithi172.stdout:+hv04.front.sepi 144.172.126.201 3 u 115 128 377 0.075 1.494 0.473 2021-02-11T06:57:10.956 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2021-02-11T06:57:18.491 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2021-02-11T06:57:54.242 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2021-02-11T06:57:54.256 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2021-02-11T06:58:05.873 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2021-02-11T06:58:08.231 INFO:teuthology.task.internal:Duration was 3266.183881 seconds 2021-02-11T06:58:08.233 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2021-02-11T06:58:08.538 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2021-02-11T06:58:08.539 DEBUG:teuthology.orchestra.run.smithi061:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2021-02-11T06:58:08.543 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2021-02-11T06:58:08.594 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2021-02-11T06:58:08.594 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi061.front.sepia.ceph.com 2021-02-11T06:58:08.594 DEBUG:teuthology.orchestra.run.smithi061:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/*.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2021-02-11T06:58:08.605 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi172.front.sepia.ceph.com 2021-02-11T06:58:08.606 DEBUG:teuthology.orchestra.run.smithi172:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/*.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2021-02-11T06:58:08.617 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2021-02-11T06:58:08.618 DEBUG:teuthology.orchestra.run.smithi061:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2021-02-11T06:58:08.652 DEBUG:teuthology.orchestra.run.smithi172:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2021-02-11T06:58:08.688 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2021-02-11T06:58:08.701 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2021-02-11T06:58:08.701 DEBUG:teuthology.orchestra.run.smithi061:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2021-02-11T06:58:08.735 DEBUG:teuthology.orchestra.run.smithi172:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2021-02-11T06:58:08.752 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2021-02-11T06:58:09.099 DEBUG:teuthology.orchestra.run.smithi061:> sudo sysctl -w kernel.core_pattern=core && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2021-02-11T06:58:09.101 DEBUG:teuthology.orchestra.run.smithi172:> sudo sysctl -w kernel.core_pattern=core && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2021-02-11T06:58:09.113 INFO:teuthology.orchestra.run.smithi061.stdout:kernel.core_pattern = core 2021-02-11T06:58:09.115 INFO:teuthology.orchestra.run.smithi172.stdout:kernel.core_pattern = core 2021-02-11T06:58:09.130 DEBUG:teuthology.orchestra.run.smithi061:> test -e /home/ubuntu/cephtest/archive/coredump 2021-02-11T06:58:09.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-11T06:58:09.179 DEBUG:teuthology.orchestra.run.smithi172:> test -e /home/ubuntu/cephtest/archive/coredump 2021-02-11T06:58:09.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-11T06:58:09.185 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2021-02-11T06:58:09.207 INFO:teuthology.task.internal:Transferring archived files... 2021-02-11T06:58:09.209 DEBUG:teuthology.misc:Transferring archived files from smithi061:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/aemerson-2021-02-11_05:45:06-rgw-wip-easter-bunny-distro-basic-smithi/5874443/remote/smithi061 2021-02-11T06:58:09.210 DEBUG:teuthology.orchestra.run.smithi061:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . 2021-02-11T06:58:09.246 DEBUG:teuthology.misc:Transferring archived files from smithi172:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/aemerson-2021-02-11_05:45:06-rgw-wip-easter-bunny-distro-basic-smithi/5874443/remote/smithi172 2021-02-11T06:58:09.247 DEBUG:teuthology.orchestra.run.smithi172:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . 2021-02-11T06:58:09.271 INFO:teuthology.task.internal:Removing archive directory... 2021-02-11T06:58:09.272 DEBUG:teuthology.orchestra.run.smithi061:> rm -rf -- /home/ubuntu/cephtest/archive 2021-02-11T06:58:09.291 DEBUG:teuthology.orchestra.run.smithi172:> rm -rf -- /home/ubuntu/cephtest/archive 2021-02-11T06:58:09.318 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2021-02-11T06:58:09.441 INFO:teuthology.task.internal:Not uploading archives. 2021-02-11T06:58:09.442 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2021-02-11T06:58:09.540 INFO:teuthology.task.internal:Tidying up after the test... 2021-02-11T06:58:09.541 DEBUG:teuthology.orchestra.run.smithi061:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2021-02-11T06:58:09.561 DEBUG:teuthology.orchestra.run.smithi172:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2021-02-11T06:58:32.361 INFO:teuthology.orchestra.run.smithi061.stdout: 524339 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 11 06:58 /home/ubuntu/cephtest 2021-02-11T06:58:32.368 INFO:teuthology.orchestra.run.smithi172.stdout: 524339 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 11 06:58 /home/ubuntu/cephtest 2021-02-11T06:58:32.370 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2021-02-11T06:58:43.065 INFO:teuthology.nuke:Checking targets against current locks 2021-02-11T06:58:43.174 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/bin/teuthology", line 33, in sys.exit(load_entry_point('teuthology', 'console_scripts', 'teuthology')()) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/scripts/run.py", line 38, in main teuthology.run.main(args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/run.py", line 412, in main report_outcome(config, archive, fake_ctx.summary, fake_ctx) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/run.py", line 269, in report_outcome nuke(fake_ctx, fake_ctx.lock) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/nuke/__init__.py", line 266, in nuke for unnuked in p: File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/nuke/__init__.py", line 280, in nuke_one keep_logs = ctx.keep_logs File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/config.py", line 255, in __getattr__ raise AttributeError(name) AttributeError: keep_logs