2021-02-11T09:33:31.925 INFO:root:teuthology version: 1.1.0-14fad317 2021-02-11T09:33:31.925 DEBUG:teuthology.run:Teuthology command: teuthology --name aemerson-2021-02-11_09:24:13-rgw-wip-blurpletastic-distro-basic-smithi --archive /home/teuthworker/archive/aemerson-2021-02-11_09:24:13-rgw-wip-blurpletastic-distro-basic-smithi/5874501 --verbose --description rgw/multisite/{clusters frontend/beast omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind} --owner scheduled_aemerson@teuthology -- /home/teuthworker/archive/aemerson-2021-02-11_09:24:13-rgw-wip-blurpletastic-distro-basic-smithi/5874501/orig.config.yaml 2021-02-11T09:33:31.959 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2021-02-11T09:33:32.012 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/aemerson-2021-02-11_09:24:13-rgw-wip-blurpletastic-distro-basic-smithi/5874501 branch: wip-blurpletastic description: rgw/multisite/{clusters frontend/beast omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind} email: aemerson@redhat.com first_in_suite: false job_id: '5874501' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: aemerson-2021-02-11_09:24:13-rgw-wip-blurpletastic-distro-basic-smithi nuke-on-error: true os_type: ubuntu overrides: admin_socket: branch: wip-blurpletastic ceph: conf: client: debug rgw: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= rgw curl low speed time: 300 rgw data log num shards: 4 rgw md log max shards: 4 rgw sync log trim interval: 0 rgw sync obj etag verify: true setgroup: ceph setuser: ceph global: osd heartbeat grace: 40 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon osd crush smoke test: false osd: debug filestore: 20 debug journal: 20 debug ms: 20 debug osd: 25 osd fast shutdown: false rocksdb delete range threshold: 0 osd.0: osd_max_omap_entries_per_request: 10 osd.1: osd_max_omap_entries_per_request: 1000 osd.2: osd_max_omap_entries_per_request: 10000 log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 5e5bb33c614118d912ca83c2e5f2be67b3acc04f wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: 5e5bb33c614118d912ca83c2e5f2be67b3acc04f rgw: compression type: random frontend: beast rgw-multisite: realm: is default: true name: test-realm zonegroups: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zonegroup zones: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zone1 - endpoints: - c2.client.0 is_default: true name: test-zone2 - endpoints: - c1.client.1 name: test-zone3 - endpoints: - c2.client.1 is_pubsub: true name: test-zone4 workunit: branch: wip-blurpletastic sha1: 5e5bb33c614118d912ca83c2e5f2be67b3acc04f owner: scheduled_aemerson@teuthology priority: 70 repo: git://git.ceph.com/ceph-ci.git roles: - - c1.mon.a - c1.mgr.x - c1.osd.0 - c1.osd.1 - c1.osd.2 - c1.client.0 - c1.client.1 - - c2.mon.a - c2.mgr.x - c2.osd.0 - c2.osd.1 - c2.osd.2 - c2.client.0 - c2.client.1 sha1: 5e5bb33c614118d912ca83c2e5f2be67b3acc04f sleep_before_teardown: 0 suite: rgw suite_branch: wip-blurpletastic suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_5e5bb33c614118d912ca83c2e5f2be67b3acc04f/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: 5e5bb33c614118d912ca83c2e5f2be67b3acc04f targets: smithi094.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDhsR+Lt7mvR59HsmPpNWxJNim8Xttba7vcp+ScC6u0Pswb6S42rmwG38WbZuU/7ep6pP2x3nzFqADW1Wnw7gA0s7j0f97KmM/1gwSTmn7wLM3vmH9YflZd0u92ejdlWXqzSlZGVQs6OeTXosJev3vWCuWPWzrqKD9PcoohB43I5OcEHwXdYw6yfox2Ev1xwUHJLfFoqPhuuTXsREJrjwfc7wnu0R/x2Uq6TVNZZGZAi4ZUjD0dDShWWpBVBBLR9CY/ld75UTGV43izFmc0s3LqjcZXoTL7Dy3PyhObcx5YlnexpyOFnC9WjKvvd8YM55slmtssLnOCuSO7f2gBX4XH smithi098.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFrxDgXyNFGHyNBYsLuHdnF6Guzsz8LugH6RDGsPAxVb7lwHj53vn1Bo+4ZTv2cbfaDSCx15SSePh5B5Bxej/1IbcypypuZmr928LW50imJaz1UqRratL3KhgE8fC6+dkFKflLidlrAGSq3WPgRGsNq1Y+wMnGK4NC9+LUHRDlGLfPOBTZcYo/SOVg/dVnubnyp+T043bk/jvhjtHNprVq1WMV0i/CyvW46Sc9NaUetAF24NIx3lxL0jxLEcDqVV4DpcpyXK2zWpNwtNxUq1erqDESFP5p/m19cA6YiJsRqwyyYjZVWdb0ZpBTK9RZAB2iO+92elpGyJchAJ4jHaSN tasks: - install: null - ceph: cluster: c1 - ceph: cluster: c2 - rgw: c1.client.0: port: 8000 c1.client.1: port: 8001 c2.client.0: port: 8000 c2.client.1: port: 8001 - rgw-multisite: null - rgw-multisite-tests: config: reconfigure_delay: 60 teuthology_branch: master timestamp: 2021-02-11_09:24:13 tube: smithi user: aemerson verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.29888 2021-02-11T09:33:32.012 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_5e5bb33c614118d912ca83c2e5f2be67b3acc04f/qa; will attempt to use it 2021-02-11T09:33:32.021 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_5e5bb33c614118d912ca83c2e5f2be67b3acc04f/qa/tasks 2021-02-11T09:33:32.022 INFO:teuthology.run_tasks:Running task internal.check_packages... 2021-02-11T09:33:32.023 INFO:teuthology.task.internal:Checking packages... 2021-02-11T09:33:32.082 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'basic' and ceph hash '5e5bb33c614118d912ca83c2e5f2be67b3acc04f' 2021-02-11T09:33:32.082 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2021-02-11T09:33:32.082 INFO:teuthology.packaging:ref: None 2021-02-11T09:33:32.083 INFO:teuthology.packaging:tag: None 2021-02-11T09:33:32.083 INFO:teuthology.packaging:branch: wip-blurpletastic 2021-02-11T09:33:32.083 INFO:teuthology.packaging:sha1: 5e5bb33c614118d912ca83c2e5f2be67b3acc04f 2021-02-11T09:33:32.083 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&ref=wip-blurpletastic 2021-02-11T09:33:32.270 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-735-g5e5bb33c-1bionic 2021-02-11T09:33:32.270 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2021-02-11T09:33:32.276 INFO:teuthology.task.internal:no buildpackages task found 2021-02-11T09:33:32.276 INFO:teuthology.run_tasks:Running task internal.save_config... 2021-02-11T09:33:36.564 INFO:teuthology.task.internal:Saving configuration 2021-02-11T09:33:36.578 INFO:teuthology.run_tasks:Running task internal.check_lock... 2021-02-11T09:33:36.593 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-11T09:33:36.613 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-02-11 09:27:57.285070', 'description': '/home/teuthworker/archive/aemerson-2021-02-11_09:24:13-rgw-wip-blurpletastic-distro-basic-smithi/5874501', 'locked_by': 'scheduled_aemerson@teuthology', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIK+AVxanScZRbUEwO71D+ehKHnWboAHmq+aFzy62FyWO', 'os_version': '18.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi094.front.sepia.ceph.com'} 2021-02-11T09:33:36.628 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-02-11 09:27:57.285051', 'description': '/home/teuthworker/archive/aemerson-2021-02-11_09:24:13-rgw-wip-blurpletastic-distro-basic-smithi/5874501', 'locked_by': 'scheduled_aemerson@teuthology', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBwmthgM/TO98zOIO1XGlLsPRY4+QMkNHQV7CtahEIBA', 'os_version': '18.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi098.front.sepia.ceph.com'} 2021-02-11T09:33:36.628 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2021-02-11T09:33:37.041 INFO:teuthology.task.internal:roles: ubuntu@smithi094.front.sepia.ceph.com - ['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'] 2021-02-11T09:33:37.041 INFO:teuthology.task.internal:roles: ubuntu@smithi098.front.sepia.ceph.com - ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1'] 2021-02-11T09:33:37.041 INFO:teuthology.run_tasks:Running task console_log... 2021-02-11T09:33:37.164 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f3859fe72f0>, signals=[15]) 2021-02-11T09:33:37.165 INFO:teuthology.run_tasks:Running task internal.connect... 2021-02-11T09:33:37.968 INFO:teuthology.task.internal:Opening connections... 2021-02-11T09:33:37.969 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi094.front.sepia.ceph.com 2021-02-11T09:33:37.972 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-02-11T09:33:38.050 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi098.front.sepia.ceph.com 2021-02-11T09:33:38.052 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-02-11T09:33:38.124 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2021-02-11T09:33:38.147 DEBUG:teuthology.orchestra.run.smithi094:> uname -m 2021-02-11T09:33:38.244 INFO:teuthology.orchestra.run.smithi094.stdout:x86_64 2021-02-11T09:33:38.245 DEBUG:teuthology.orchestra.run.smithi094:> cat /etc/os-release 2021-02-11T09:33:38.290 INFO:teuthology.orchestra.run.smithi094.stdout:NAME="Ubuntu" 2021-02-11T09:33:38.290 INFO:teuthology.orchestra.run.smithi094.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2021-02-11T09:33:38.290 INFO:teuthology.orchestra.run.smithi094.stdout:ID=ubuntu 2021-02-11T09:33:38.290 INFO:teuthology.orchestra.run.smithi094.stdout:ID_LIKE=debian 2021-02-11T09:33:38.291 INFO:teuthology.orchestra.run.smithi094.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2021-02-11T09:33:38.291 INFO:teuthology.orchestra.run.smithi094.stdout:VERSION_ID="18.04" 2021-02-11T09:33:38.291 INFO:teuthology.orchestra.run.smithi094.stdout:HOME_URL="https://www.ubuntu.com/" 2021-02-11T09:33:38.291 INFO:teuthology.orchestra.run.smithi094.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-02-11T09:33:38.291 INFO:teuthology.orchestra.run.smithi094.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-02-11T09:33:38.291 INFO:teuthology.orchestra.run.smithi094.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-02-11T09:33:38.291 INFO:teuthology.orchestra.run.smithi094.stdout:VERSION_CODENAME=bionic 2021-02-11T09:33:38.291 INFO:teuthology.orchestra.run.smithi094.stdout:UBUNTU_CODENAME=bionic 2021-02-11T09:33:38.292 INFO:teuthology.lock.ops:Updating smithi094.front.sepia.ceph.com on lock server 2021-02-11T09:33:38.307 DEBUG:teuthology.orchestra.run.smithi098:> uname -m 2021-02-11T09:33:38.328 INFO:teuthology.orchestra.run.smithi098.stdout:x86_64 2021-02-11T09:33:38.328 DEBUG:teuthology.orchestra.run.smithi098:> cat /etc/os-release 2021-02-11T09:33:38.374 INFO:teuthology.orchestra.run.smithi098.stdout:NAME="Ubuntu" 2021-02-11T09:33:38.374 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2021-02-11T09:33:38.374 INFO:teuthology.orchestra.run.smithi098.stdout:ID=ubuntu 2021-02-11T09:33:38.374 INFO:teuthology.orchestra.run.smithi098.stdout:ID_LIKE=debian 2021-02-11T09:33:38.374 INFO:teuthology.orchestra.run.smithi098.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2021-02-11T09:33:38.374 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION_ID="18.04" 2021-02-11T09:33:38.374 INFO:teuthology.orchestra.run.smithi098.stdout:HOME_URL="https://www.ubuntu.com/" 2021-02-11T09:33:38.374 INFO:teuthology.orchestra.run.smithi098.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-02-11T09:33:38.374 INFO:teuthology.orchestra.run.smithi098.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-02-11T09:33:38.374 INFO:teuthology.orchestra.run.smithi098.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-02-11T09:33:38.374 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION_CODENAME=bionic 2021-02-11T09:33:38.374 INFO:teuthology.orchestra.run.smithi098.stdout:UBUNTU_CODENAME=bionic 2021-02-11T09:33:38.375 INFO:teuthology.lock.ops:Updating smithi098.front.sepia.ceph.com on lock server 2021-02-11T09:33:38.388 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2021-02-11T09:33:38.398 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2021-02-11T09:33:38.790 INFO:teuthology.task.internal:Checking for old test directory... 2021-02-11T09:33:38.790 DEBUG:teuthology.orchestra.run.smithi094:> test '!' -e /home/ubuntu/cephtest 2021-02-11T09:33:38.792 DEBUG:teuthology.orchestra.run.smithi098:> test '!' -e /home/ubuntu/cephtest 2021-02-11T09:33:38.797 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2021-02-11T09:33:38.888 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2021-02-11T09:33:38.889 DEBUG:teuthology.orchestra.run.smithi094:> test -z $(ls -A /var/lib/ceph) 2021-02-11T09:33:38.891 DEBUG:teuthology.orchestra.run.smithi098:> test -z $(ls -A /var/lib/ceph) 2021-02-11T09:33:38.909 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2021-02-11T09:33:38.992 INFO:teuthology.run_tasks:Running task kernel... 2021-02-11T09:33:39.009 DEBUG:teuthology.task.kernel:normalized config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c1.mgr.x': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.0': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.1': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.2': {'kdb': True, 'sha1': 'distro'}, 'c1.client.0': {'kdb': True, 'sha1': 'distro'}, 'c1.client.1': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mgr.x': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.0': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.1': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.2': {'kdb': True, 'sha1': 'distro'}, 'c2.client.0': {'kdb': True, 'sha1': 'distro'}, 'c2.client.1': {'kdb': True, 'sha1': 'distro'}} 2021-02-11T09:33:39.009 INFO:teuthology.task.kernel:config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2021-02-11T09:33:39.009 DEBUG:teuthology.orchestra.run.smithi094:> uname -r 2021-02-11T09:33:39.015 INFO:teuthology.orchestra.run.smithi094.stdout:4.15.0-134-generic 2021-02-11T09:33:39.015 INFO:teuthology.task.kernel:Running kernel on smithi094: 4.15.0-134-generic 2021-02-11T09:33:39.016 DEBUG:teuthology.orchestra.run.smithi094:> sudo apt-get clean 2021-02-11T09:33:39.118 DEBUG:teuthology.orchestra.run.smithi094:> sudo apt-get update 2021-02-11T09:33:39.407 INFO:teuthology.orchestra.run.smithi094.stdout:Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease 2021-02-11T09:33:39.407 INFO:teuthology.orchestra.run.smithi094.stdout:Get:2 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2021-02-11T09:33:39.466 INFO:teuthology.orchestra.run.smithi094.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2021-02-11T09:33:39.767 INFO:teuthology.orchestra.run.smithi094.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2021-02-11T09:33:39.821 INFO:teuthology.orchestra.run.smithi094.stdout:Get:5 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [1,543 kB] 2021-02-11T09:33:39.955 INFO:teuthology.orchestra.run.smithi094.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages [1,218 kB] 2021-02-11T09:33:40.285 INFO:teuthology.orchestra.run.smithi094.stdout:Get:7 http://security.ubuntu.com/ubuntu bionic-security/main i386 Packages [915 kB] 2021-02-11T09:33:40.301 INFO:teuthology.orchestra.run.smithi094.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [1,883 kB] 2021-02-11T09:33:40.320 INFO:teuthology.orchestra.run.smithi094.stdout:Get:9 http://security.ubuntu.com/ubuntu bionic-security/main Translation-en [298 kB] 2021-02-11T09:33:40.330 INFO:teuthology.orchestra.run.smithi094.stdout:Get:10 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [226 kB] 2021-02-11T09:33:40.339 INFO:teuthology.orchestra.run.smithi094.stdout:Get:11 http://security.ubuntu.com/ubuntu bionic-security/restricted Translation-en [29.9 kB] 2021-02-11T09:33:40.340 INFO:teuthology.orchestra.run.smithi094.stdout:Get:12 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1,109 kB] 2021-02-11T09:33:40.402 INFO:teuthology.orchestra.run.smithi094.stdout:Get:13 http://security.ubuntu.com/ubuntu bionic-security/universe i386 Packages [975 kB] 2021-02-11T09:33:40.403 INFO:teuthology.orchestra.run.smithi094.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic-updates/main Translation-en [390 kB] 2021-02-11T09:33:40.419 INFO:teuthology.orchestra.run.smithi094.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [247 kB] 2021-02-11T09:33:40.427 INFO:teuthology.orchestra.run.smithi094.stdout:Get:16 http://security.ubuntu.com/ubuntu bionic-security/universe Translation-en [248 kB] 2021-02-11T09:33:40.429 INFO:teuthology.orchestra.run.smithi094.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic-updates/restricted Translation-en [33.4 kB] 2021-02-11T09:33:40.430 INFO:teuthology.orchestra.run.smithi094.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1,718 kB] 2021-02-11T09:33:40.473 INFO:teuthology.orchestra.run.smithi094.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic-updates/universe i386 Packages [1,560 kB] 2021-02-11T09:33:40.522 INFO:teuthology.orchestra.run.smithi094.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic-updates/universe Translation-en [363 kB] 2021-02-11T09:33:40.529 INFO:teuthology.orchestra.run.smithi094.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse i386 Packages [14.3 kB] 2021-02-11T09:33:40.530 INFO:teuthology.orchestra.run.smithi094.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [31.8 kB] 2021-02-11T09:33:40.530 INFO:teuthology.orchestra.run.smithi094.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse Translation-en [7,300 B] 2021-02-11T09:33:42.062 INFO:teuthology.orchestra.run.smithi094.stdout:Fetched 13.1 MB in 3s (4,971 kB/s) 2021-02-11T09:33:43.486 INFO:teuthology.orchestra.run.smithi094.stdout:Reading package lists... 2021-02-11T09:33:43.534 DEBUG:teuthology.orchestra.run.smithi094:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2021-02-11T09:33:43.608 INFO:teuthology.orchestra.run.smithi094.stdout:Reading package lists... 2021-02-11T09:33:43.855 INFO:teuthology.orchestra.run.smithi094.stdout:Building dependency tree... 2021-02-11T09:33:43.856 INFO:teuthology.orchestra.run.smithi094.stdout:Reading state information... 2021-02-11T09:33:43.934 INFO:teuthology.orchestra.run.smithi094.stderr:E: Unable to locate package linux-image-current-generic 2021-02-11T09:33:43.936 DEBUG:teuthology.orchestra.run:got remote process result: 100 2021-02-11T09:33:43.936 DEBUG:teuthology.orchestra.run.smithi094:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2021-02-11T09:33:44.013 INFO:teuthology.orchestra.run.smithi094.stdout:Reading package lists... 2021-02-11T09:33:44.250 INFO:teuthology.orchestra.run.smithi094.stdout:Building dependency tree... 2021-02-11T09:33:44.251 INFO:teuthology.orchestra.run.smithi094.stdout:Reading state information... 2021-02-11T09:33:44.424 INFO:teuthology.orchestra.run.smithi094.stdout:The following additional packages will be installed: 2021-02-11T09:33:44.424 INFO:teuthology.orchestra.run.smithi094.stdout: linux-generic linux-headers-4.15.0-135 linux-headers-4.15.0-135-generic 2021-02-11T09:33:44.424 INFO:teuthology.orchestra.run.smithi094.stdout: linux-headers-generic linux-image-4.15.0-135-generic 2021-02-11T09:33:44.424 INFO:teuthology.orchestra.run.smithi094.stdout: linux-modules-4.15.0-135-generic linux-modules-extra-4.15.0-135-generic 2021-02-11T09:33:44.426 INFO:teuthology.orchestra.run.smithi094.stdout:Suggested packages: 2021-02-11T09:33:44.426 INFO:teuthology.orchestra.run.smithi094.stdout: fdutils linux-doc-4.15.0 | linux-source-4.15.0 linux-tools 2021-02-11T09:33:44.426 INFO:teuthology.orchestra.run.smithi094.stdout:Recommended packages: 2021-02-11T09:33:44.426 INFO:teuthology.orchestra.run.smithi094.stdout: thermald 2021-02-11T09:33:44.459 INFO:teuthology.orchestra.run.smithi094.stdout:The following NEW packages will be installed: 2021-02-11T09:33:44.459 INFO:teuthology.orchestra.run.smithi094.stdout: linux-headers-4.15.0-135 linux-headers-4.15.0-135-generic 2021-02-11T09:33:44.459 INFO:teuthology.orchestra.run.smithi094.stdout: linux-image-4.15.0-135-generic linux-modules-4.15.0-135-generic 2021-02-11T09:33:44.459 INFO:teuthology.orchestra.run.smithi094.stdout: linux-modules-extra-4.15.0-135-generic 2021-02-11T09:33:44.461 INFO:teuthology.orchestra.run.smithi094.stdout:The following packages will be upgraded: 2021-02-11T09:33:44.461 INFO:teuthology.orchestra.run.smithi094.stdout: linux-generic linux-headers-generic linux-image-generic 2021-02-11T09:33:44.640 INFO:teuthology.orchestra.run.smithi094.stdout:3 upgraded, 5 newly installed, 0 to remove and 23 not upgraded. 2021-02-11T09:33:44.641 INFO:teuthology.orchestra.run.smithi094.stdout:Need to get 66.0 MB of archives. 2021-02-11T09:33:44.641 INFO:teuthology.orchestra.run.smithi094.stdout:After this operation, 334 MB of additional disk space will be used. 2021-02-11T09:33:44.641 INFO:teuthology.orchestra.run.smithi094.stdout:Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-4.15.0-135-generic amd64 4.15.0-135.139 [13.1 MB] 2021-02-11T09:33:45.654 INFO:teuthology.orchestra.run.smithi094.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-4.15.0-135-generic amd64 4.15.0-135.139 [8,019 kB] 2021-02-11T09:33:45.854 INFO:teuthology.orchestra.run.smithi094.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-extra-4.15.0-135-generic amd64 4.15.0-135.139 [32.8 MB] 2021-02-11T09:33:46.879 INFO:teuthology.orchestra.run.smithi094.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-generic amd64 4.15.0.135.122 [1,868 B] 2021-02-11T09:33:46.879 INFO:teuthology.orchestra.run.smithi094.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-generic amd64 4.15.0.135.122 [2,592 B] 2021-02-11T09:33:46.879 INFO:teuthology.orchestra.run.smithi094.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-135 all 4.15.0-135.139 [10.9 MB] 2021-02-11T09:33:47.198 INFO:teuthology.orchestra.run.smithi094.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-135-generic amd64 4.15.0-135.139 [1,097 kB] 2021-02-11T09:33:47.215 INFO:teuthology.orchestra.run.smithi094.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-generic amd64 4.15.0.135.122 [2,488 B] 2021-02-11T09:33:47.716 INFO:teuthology.orchestra.run.smithi094.stdout:Fetched 66.0 MB in 3s (24.1 MB/s) 2021-02-11T09:33:47.877 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package linux-modules-4.15.0-135-generic. 2021-02-11T09:33:47.914 INFO:teuthology.orchestra.run.smithi094.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102079 files and directories currently installed.) 2021-02-11T09:33:47.917 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../0-linux-modules-4.15.0-135-generic_4.15.0-135.139_amd64.deb ... 2021-02-11T09:33:47.957 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking linux-modules-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-11T09:33:49.651 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package linux-image-4.15.0-135-generic. 2021-02-11T09:33:49.664 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../1-linux-image-4.15.0-135-generic_4.15.0-135.139_amd64.deb ... 2021-02-11T09:33:49.762 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking linux-image-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-11T09:33:50.263 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package linux-modules-extra-4.15.0-135-generic. 2021-02-11T09:33:50.275 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../2-linux-modules-extra-4.15.0-135-generic_4.15.0-135.139_amd64.deb ... 2021-02-11T09:33:50.313 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking linux-modules-extra-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-11T09:33:54.057 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../3-linux-generic_4.15.0.135.122_amd64.deb ... 2021-02-11T09:33:54.177 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking linux-generic (4.15.0.135.122) over (4.15.0.134.121) ... 2021-02-11T09:33:54.485 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../4-linux-image-generic_4.15.0.135.122_amd64.deb ... 2021-02-11T09:33:54.607 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking linux-image-generic (4.15.0.135.122) over (4.15.0.134.121) ... 2021-02-11T09:33:54.943 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package linux-headers-4.15.0-135. 2021-02-11T09:33:54.955 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../5-linux-headers-4.15.0-135_4.15.0-135.139_all.deb ... 2021-02-11T09:33:54.993 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking linux-headers-4.15.0-135 (4.15.0-135.139) ... 2021-02-11T09:33:59.551 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package linux-headers-4.15.0-135-generic. 2021-02-11T09:33:59.572 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../6-linux-headers-4.15.0-135-generic_4.15.0-135.139_amd64.deb ... 2021-02-11T09:33:59.609 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking linux-headers-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-11T09:34:02.527 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../7-linux-headers-generic_4.15.0.135.122_amd64.deb ... 2021-02-11T09:34:02.652 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking linux-headers-generic (4.15.0.135.122) over (4.15.0.134.121) ... 2021-02-11T09:34:02.986 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up linux-modules-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-11T09:34:03.524 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up linux-headers-4.15.0-135 (4.15.0-135.139) ... 2021-02-11T09:34:03.658 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up linux-image-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-11T09:34:04.103 INFO:teuthology.orchestra.run.smithi094.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-4.15.0-135-generic 2021-02-11T09:34:04.103 INFO:teuthology.orchestra.run.smithi094.stdout:I: /initrd.img is now a symlink to boot/initrd.img-4.15.0-135-generic 2021-02-11T09:34:04.235 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up linux-modules-extra-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-11T09:34:04.728 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up linux-headers-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-11T09:34:04.837 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up linux-image-generic (4.15.0.135.122) ... 2021-02-11T09:34:04.955 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up linux-headers-generic (4.15.0.135.122) ... 2021-02-11T09:34:05.081 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up linux-generic (4.15.0.135.122) ... 2021-02-11T09:34:12.117 INFO:teuthology.orchestra.run.smithi094.stdout:Processing triggers for linux-image-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-11T09:34:12.117 INFO:teuthology.orchestra.run.smithi094.stdout:/etc/kernel/postinst.d/initramfs-tools: 2021-02-11T09:34:12.117 INFO:teuthology.orchestra.run.smithi094.stdout:update-initramfs: Generating /boot/initrd.img-4.15.0-135-generic 2021-02-11T09:34:20.423 INFO:teuthology.orchestra.run.smithi094.stdout:/etc/kernel/postinst.d/zz-update-grub: 2021-02-11T09:34:20.424 INFO:teuthology.orchestra.run.smithi094.stdout:Sourcing file `/etc/default/grub' 2021-02-11T09:34:20.460 INFO:teuthology.orchestra.run.smithi094.stdout:Generating grub configuration file ... 2021-02-11T09:34:20.824 INFO:teuthology.orchestra.run.smithi094.stdout:Found linux image: /boot/vmlinuz-4.15.0-135-generic 2021-02-11T09:34:20.857 INFO:teuthology.orchestra.run.smithi094.stdout:Found initrd image: /boot/initrd.img-4.15.0-135-generic 2021-02-11T09:34:21.209 INFO:teuthology.orchestra.run.smithi094.stdout:Found linux image: /boot/vmlinuz-4.15.0-134-generic 2021-02-11T09:34:21.244 INFO:teuthology.orchestra.run.smithi094.stdout:Found initrd image: /boot/initrd.img-4.15.0-134-generic 2021-02-11T09:34:22.374 INFO:teuthology.orchestra.run.smithi094.stdout:done 2021-02-11T09:34:22.788 DEBUG:teuthology.orchestra.run.smithi094:> dpkg -s linux-image-generic 2021-02-11T09:34:22.850 INFO:teuthology.orchestra.run.smithi094.stdout:Package: linux-image-generic 2021-02-11T09:34:22.850 INFO:teuthology.orchestra.run.smithi094.stdout:Status: install ok installed 2021-02-11T09:34:22.850 INFO:teuthology.orchestra.run.smithi094.stdout:Priority: optional 2021-02-11T09:34:22.850 INFO:teuthology.orchestra.run.smithi094.stdout:Section: kernel 2021-02-11T09:34:22.850 INFO:teuthology.orchestra.run.smithi094.stdout:Installed-Size: 17 2021-02-11T09:34:22.850 INFO:teuthology.orchestra.run.smithi094.stdout:Maintainer: Ubuntu Kernel Team 2021-02-11T09:34:22.851 INFO:teuthology.orchestra.run.smithi094.stdout:Architecture: amd64 2021-02-11T09:34:22.851 INFO:teuthology.orchestra.run.smithi094.stdout:Source: linux-meta 2021-02-11T09:34:22.851 INFO:teuthology.orchestra.run.smithi094.stdout:Version: 4.15.0.135.122 2021-02-11T09:34:22.851 INFO:teuthology.orchestra.run.smithi094.stdout:Provides: wireguard-modules (= 1.0.20201112-1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.10) 2021-02-11T09:34:22.851 INFO:teuthology.orchestra.run.smithi094.stdout:Depends: linux-image-4.15.0-135-generic, linux-modules-extra-4.15.0-135-generic, linux-firmware, intel-microcode, amd64-microcode 2021-02-11T09:34:22.851 INFO:teuthology.orchestra.run.smithi094.stdout:Recommends: thermald 2021-02-11T09:34:22.851 INFO:teuthology.orchestra.run.smithi094.stdout:Description: Generic Linux kernel image 2021-02-11T09:34:22.851 INFO:teuthology.orchestra.run.smithi094.stdout: This package will always depend on the latest generic kernel image 2021-02-11T09:34:22.851 INFO:teuthology.orchestra.run.smithi094.stdout: available. 2021-02-11T09:34:22.851 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.15.0-134-generic Expected: 4.15.0-135-generic 2021-02-11T09:34:22.852 DEBUG:teuthology.orchestra.run.smithi098:> uname -r 2021-02-11T09:34:22.858 INFO:teuthology.orchestra.run.smithi098.stdout:4.15.0-134-generic 2021-02-11T09:34:22.858 INFO:teuthology.task.kernel:Running kernel on smithi098: 4.15.0-134-generic 2021-02-11T09:34:22.858 DEBUG:teuthology.orchestra.run.smithi098:> sudo apt-get clean 2021-02-11T09:34:22.974 DEBUG:teuthology.orchestra.run.smithi098:> sudo apt-get update 2021-02-11T09:34:23.240 INFO:teuthology.orchestra.run.smithi098.stdout:Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease 2021-02-11T09:34:23.240 INFO:teuthology.orchestra.run.smithi098.stdout:Get:2 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2021-02-11T09:34:23.297 INFO:teuthology.orchestra.run.smithi098.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2021-02-11T09:34:23.566 INFO:teuthology.orchestra.run.smithi098.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2021-02-11T09:34:23.640 INFO:teuthology.orchestra.run.smithi098.stdout:Get:5 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [1,543 kB] 2021-02-11T09:34:23.776 INFO:teuthology.orchestra.run.smithi098.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [1,883 kB] 2021-02-11T09:34:24.075 INFO:teuthology.orchestra.run.smithi098.stdout:Get:7 http://security.ubuntu.com/ubuntu bionic-security/main i386 Packages [915 kB] 2021-02-11T09:34:24.095 INFO:teuthology.orchestra.run.smithi098.stdout:Get:8 http://security.ubuntu.com/ubuntu bionic-security/main Translation-en [298 kB] 2021-02-11T09:34:24.100 INFO:teuthology.orchestra.run.smithi098.stdout:Get:9 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [226 kB] 2021-02-11T09:34:24.105 INFO:teuthology.orchestra.run.smithi098.stdout:Get:10 http://security.ubuntu.com/ubuntu bionic-security/restricted Translation-en [29.9 kB] 2021-02-11T09:34:24.106 INFO:teuthology.orchestra.run.smithi098.stdout:Get:11 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1,109 kB] 2021-02-11T09:34:24.173 INFO:teuthology.orchestra.run.smithi098.stdout:Get:12 http://security.ubuntu.com/ubuntu bionic-security/universe i386 Packages [975 kB] 2021-02-11T09:34:24.191 INFO:teuthology.orchestra.run.smithi098.stdout:Get:13 http://archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages [1,218 kB] 2021-02-11T09:34:24.198 INFO:teuthology.orchestra.run.smithi098.stdout:Get:14 http://security.ubuntu.com/ubuntu bionic-security/universe Translation-en [248 kB] 2021-02-11T09:34:24.211 INFO:teuthology.orchestra.run.smithi098.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic-updates/main Translation-en [390 kB] 2021-02-11T09:34:24.219 INFO:teuthology.orchestra.run.smithi098.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [247 kB] 2021-02-11T09:34:24.272 INFO:teuthology.orchestra.run.smithi098.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic-updates/restricted Translation-en [33.4 kB] 2021-02-11T09:34:24.273 INFO:teuthology.orchestra.run.smithi098.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1,718 kB] 2021-02-11T09:34:24.303 INFO:teuthology.orchestra.run.smithi098.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic-updates/universe i386 Packages [1,560 kB] 2021-02-11T09:34:24.355 INFO:teuthology.orchestra.run.smithi098.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic-updates/universe Translation-en [363 kB] 2021-02-11T09:34:24.362 INFO:teuthology.orchestra.run.smithi098.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [31.8 kB] 2021-02-11T09:34:24.363 INFO:teuthology.orchestra.run.smithi098.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse i386 Packages [14.3 kB] 2021-02-11T09:34:24.363 INFO:teuthology.orchestra.run.smithi098.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse Translation-en [7,300 B] 2021-02-11T09:34:25.762 INFO:teuthology.orchestra.run.smithi098.stdout:Fetched 13.1 MB in 3s (5,089 kB/s) 2021-02-11T09:34:27.191 INFO:teuthology.orchestra.run.smithi098.stdout:Reading package lists... 2021-02-11T09:34:27.238 DEBUG:teuthology.orchestra.run.smithi098:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2021-02-11T09:34:27.361 INFO:teuthology.orchestra.run.smithi098.stdout:Reading package lists... 2021-02-11T09:34:27.601 INFO:teuthology.orchestra.run.smithi098.stdout:Building dependency tree... 2021-02-11T09:34:27.601 INFO:teuthology.orchestra.run.smithi098.stdout:Reading state information... 2021-02-11T09:34:27.680 INFO:teuthology.orchestra.run.smithi098.stderr:E: Unable to locate package linux-image-current-generic 2021-02-11T09:34:27.682 DEBUG:teuthology.orchestra.run:got remote process result: 100 2021-02-11T09:34:27.683 DEBUG:teuthology.orchestra.run.smithi098:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2021-02-11T09:34:27.759 INFO:teuthology.orchestra.run.smithi098.stdout:Reading package lists... 2021-02-11T09:34:27.988 INFO:teuthology.orchestra.run.smithi098.stdout:Building dependency tree... 2021-02-11T09:34:27.989 INFO:teuthology.orchestra.run.smithi098.stdout:Reading state information... 2021-02-11T09:34:28.162 INFO:teuthology.orchestra.run.smithi098.stdout:The following additional packages will be installed: 2021-02-11T09:34:28.162 INFO:teuthology.orchestra.run.smithi098.stdout: linux-generic linux-headers-4.15.0-135 linux-headers-4.15.0-135-generic 2021-02-11T09:34:28.162 INFO:teuthology.orchestra.run.smithi098.stdout: linux-headers-generic linux-image-4.15.0-135-generic 2021-02-11T09:34:28.162 INFO:teuthology.orchestra.run.smithi098.stdout: linux-modules-4.15.0-135-generic linux-modules-extra-4.15.0-135-generic 2021-02-11T09:34:28.164 INFO:teuthology.orchestra.run.smithi098.stdout:Suggested packages: 2021-02-11T09:34:28.164 INFO:teuthology.orchestra.run.smithi098.stdout: fdutils linux-doc-4.15.0 | linux-source-4.15.0 linux-tools 2021-02-11T09:34:28.164 INFO:teuthology.orchestra.run.smithi098.stdout:Recommended packages: 2021-02-11T09:34:28.164 INFO:teuthology.orchestra.run.smithi098.stdout: thermald 2021-02-11T09:34:28.198 INFO:teuthology.orchestra.run.smithi098.stdout:The following NEW packages will be installed: 2021-02-11T09:34:28.198 INFO:teuthology.orchestra.run.smithi098.stdout: linux-headers-4.15.0-135 linux-headers-4.15.0-135-generic 2021-02-11T09:34:28.198 INFO:teuthology.orchestra.run.smithi098.stdout: linux-image-4.15.0-135-generic linux-modules-4.15.0-135-generic 2021-02-11T09:34:28.198 INFO:teuthology.orchestra.run.smithi098.stdout: linux-modules-extra-4.15.0-135-generic 2021-02-11T09:34:28.200 INFO:teuthology.orchestra.run.smithi098.stdout:The following packages will be upgraded: 2021-02-11T09:34:28.200 INFO:teuthology.orchestra.run.smithi098.stdout: linux-generic linux-headers-generic linux-image-generic 2021-02-11T09:34:28.374 INFO:teuthology.orchestra.run.smithi098.stdout:3 upgraded, 5 newly installed, 0 to remove and 23 not upgraded. 2021-02-11T09:34:28.374 INFO:teuthology.orchestra.run.smithi098.stdout:Need to get 66.0 MB of archives. 2021-02-11T09:34:28.374 INFO:teuthology.orchestra.run.smithi098.stdout:After this operation, 334 MB of additional disk space will be used. 2021-02-11T09:34:28.374 INFO:teuthology.orchestra.run.smithi098.stdout:Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-4.15.0-135-generic amd64 4.15.0-135.139 [13.1 MB] 2021-02-11T09:34:29.358 INFO:teuthology.orchestra.run.smithi098.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-4.15.0-135-generic amd64 4.15.0-135.139 [8,019 kB] 2021-02-11T09:34:29.551 INFO:teuthology.orchestra.run.smithi098.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-extra-4.15.0-135-generic amd64 4.15.0-135.139 [32.8 MB] 2021-02-11T09:34:30.487 INFO:teuthology.orchestra.run.smithi098.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-generic amd64 4.15.0.135.122 [1,868 B] 2021-02-11T09:34:30.487 INFO:teuthology.orchestra.run.smithi098.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-generic amd64 4.15.0.135.122 [2,592 B] 2021-02-11T09:34:30.487 INFO:teuthology.orchestra.run.smithi098.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-135 all 4.15.0-135.139 [10.9 MB] 2021-02-11T09:34:30.794 INFO:teuthology.orchestra.run.smithi098.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-135-generic amd64 4.15.0-135.139 [1,097 kB] 2021-02-11T09:34:30.847 INFO:teuthology.orchestra.run.smithi098.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-generic amd64 4.15.0.135.122 [2,488 B] 2021-02-11T09:34:31.307 INFO:teuthology.orchestra.run.smithi098.stdout:Fetched 66.0 MB in 3s (25.1 MB/s) 2021-02-11T09:34:31.493 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package linux-modules-4.15.0-135-generic. 2021-02-11T09:34:31.533 INFO:teuthology.orchestra.run.smithi098.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102079 files and directories currently installed.) 2021-02-11T09:34:31.535 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../0-linux-modules-4.15.0-135-generic_4.15.0-135.139_amd64.deb ... 2021-02-11T09:34:31.582 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking linux-modules-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-11T09:34:33.158 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package linux-image-4.15.0-135-generic. 2021-02-11T09:34:33.172 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../1-linux-image-4.15.0-135-generic_4.15.0-135.139_amd64.deb ... 2021-02-11T09:34:33.274 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking linux-image-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-11T09:34:33.812 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package linux-modules-extra-4.15.0-135-generic. 2021-02-11T09:34:33.822 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../2-linux-modules-extra-4.15.0-135-generic_4.15.0-135.139_amd64.deb ... 2021-02-11T09:34:33.861 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking linux-modules-extra-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-11T09:34:37.575 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../3-linux-generic_4.15.0.135.122_amd64.deb ... 2021-02-11T09:34:42.012 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking linux-generic (4.15.0.135.122) over (4.15.0.134.121) ... 2021-02-11T09:34:42.012 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../4-linux-image-generic_4.15.0.135.122_amd64.deb ... 2021-02-11T09:34:42.012 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking linux-image-generic (4.15.0.135.122) over (4.15.0.134.121) ... 2021-02-11T09:34:42.013 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package linux-headers-4.15.0-135. 2021-02-11T09:34:42.013 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../5-linux-headers-4.15.0-135_4.15.0-135.139_all.deb ... 2021-02-11T09:34:42.013 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking linux-headers-4.15.0-135 (4.15.0-135.139) ... 2021-02-11T09:34:43.270 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package linux-headers-4.15.0-135-generic. 2021-02-11T09:34:43.291 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../6-linux-headers-4.15.0-135-generic_4.15.0-135.139_amd64.deb ... 2021-02-11T09:34:43.328 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking linux-headers-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-11T09:34:45.746 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../7-linux-headers-generic_4.15.0.135.122_amd64.deb ... 2021-02-11T09:34:45.885 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking linux-headers-generic (4.15.0.135.122) over (4.15.0.134.121) ... 2021-02-11T09:34:46.212 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up linux-modules-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-11T09:34:46.766 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up linux-headers-4.15.0-135 (4.15.0-135.139) ... 2021-02-11T09:34:46.884 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up linux-image-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-11T09:34:47.329 INFO:teuthology.orchestra.run.smithi098.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-4.15.0-135-generic 2021-02-11T09:34:47.329 INFO:teuthology.orchestra.run.smithi098.stdout:I: /initrd.img is now a symlink to boot/initrd.img-4.15.0-135-generic 2021-02-11T09:34:47.461 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up linux-modules-extra-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-11T09:34:47.980 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up linux-headers-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-11T09:34:48.106 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up linux-image-generic (4.15.0.135.122) ... 2021-02-11T09:34:48.232 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up linux-headers-generic (4.15.0.135.122) ... 2021-02-11T09:34:48.349 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up linux-generic (4.15.0.135.122) ... 2021-02-11T09:34:48.459 INFO:teuthology.orchestra.run.smithi098.stdout:Processing triggers for linux-image-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-11T09:34:48.665 INFO:teuthology.orchestra.run.smithi098.stdout:/etc/kernel/postinst.d/initramfs-tools: 2021-02-11T09:34:48.665 INFO:teuthology.orchestra.run.smithi098.stdout:update-initramfs: Generating /boot/initrd.img-4.15.0-135-generic 2021-02-11T09:35:04.334 INFO:teuthology.orchestra.run.smithi098.stdout:/etc/kernel/postinst.d/zz-update-grub: 2021-02-11T09:35:04.335 INFO:teuthology.orchestra.run.smithi098.stdout:Sourcing file `/etc/default/grub' 2021-02-11T09:35:04.367 INFO:teuthology.orchestra.run.smithi098.stdout:Generating grub configuration file ... 2021-02-11T09:35:04.731 INFO:teuthology.orchestra.run.smithi098.stdout:Found linux image: /boot/vmlinuz-4.15.0-135-generic 2021-02-11T09:35:04.780 INFO:teuthology.orchestra.run.smithi098.stdout:Found initrd image: /boot/initrd.img-4.15.0-135-generic 2021-02-11T09:35:05.136 INFO:teuthology.orchestra.run.smithi098.stdout:Found linux image: /boot/vmlinuz-4.15.0-134-generic 2021-02-11T09:35:05.168 INFO:teuthology.orchestra.run.smithi098.stdout:Found initrd image: /boot/initrd.img-4.15.0-134-generic 2021-02-11T09:35:06.660 INFO:teuthology.orchestra.run.smithi098.stdout:done 2021-02-11T09:35:07.086 DEBUG:teuthology.orchestra.run.smithi098:> dpkg -s linux-image-generic 2021-02-11T09:35:07.109 INFO:teuthology.orchestra.run.smithi098.stdout:Package: linux-image-generic 2021-02-11T09:35:07.109 INFO:teuthology.orchestra.run.smithi098.stdout:Status: install ok installed 2021-02-11T09:35:07.109 INFO:teuthology.orchestra.run.smithi098.stdout:Priority: optional 2021-02-11T09:35:07.109 INFO:teuthology.orchestra.run.smithi098.stdout:Section: kernel 2021-02-11T09:35:07.109 INFO:teuthology.orchestra.run.smithi098.stdout:Installed-Size: 17 2021-02-11T09:35:07.109 INFO:teuthology.orchestra.run.smithi098.stdout:Maintainer: Ubuntu Kernel Team 2021-02-11T09:35:07.109 INFO:teuthology.orchestra.run.smithi098.stdout:Architecture: amd64 2021-02-11T09:35:07.109 INFO:teuthology.orchestra.run.smithi098.stdout:Source: linux-meta 2021-02-11T09:35:07.109 INFO:teuthology.orchestra.run.smithi098.stdout:Version: 4.15.0.135.122 2021-02-11T09:35:07.109 INFO:teuthology.orchestra.run.smithi098.stdout:Provides: wireguard-modules (= 1.0.20201112-1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.10) 2021-02-11T09:35:07.109 INFO:teuthology.orchestra.run.smithi098.stdout:Depends: linux-image-4.15.0-135-generic, linux-modules-extra-4.15.0-135-generic, linux-firmware, intel-microcode, amd64-microcode 2021-02-11T09:35:07.110 INFO:teuthology.orchestra.run.smithi098.stdout:Recommends: thermald 2021-02-11T09:35:07.110 INFO:teuthology.orchestra.run.smithi098.stdout:Description: Generic Linux kernel image 2021-02-11T09:35:07.110 INFO:teuthology.orchestra.run.smithi098.stdout: This package will always depend on the latest generic kernel image 2021-02-11T09:35:07.110 INFO:teuthology.orchestra.run.smithi098.stdout: available. 2021-02-11T09:35:07.110 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.15.0-134-generic Expected: 4.15.0-135-generic 2021-02-11T09:35:07.110 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2021-02-11T09:35:07.110 DEBUG:teuthology.task.kernel:src is distro, skipping download 2021-02-11T09:35:07.110 DEBUG:teuthology.task.kernel:src is distro, skipping download 2021-02-11T09:35:07.110 INFO:teuthology.task.kernel:Installing distro kernel on c1.mon.a... 2021-02-11T09:35:07.111 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi094.front.sepia.ceph.com, path=None, version=distro) 2021-02-11T09:35:07.111 DEBUG:teuthology.orchestra.run.smithi094:> sudo apt-get clean 2021-02-11T09:35:07.217 DEBUG:teuthology.orchestra.run.smithi094:> sudo apt-get update 2021-02-11T09:35:07.347 INFO:teuthology.orchestra.run.smithi094.stdout:Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease 2021-02-11T09:35:07.454 INFO:teuthology.orchestra.run.smithi094.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2021-02-11T09:35:07.534 INFO:teuthology.orchestra.run.smithi094.stdout:Hit:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2021-02-11T09:35:07.614 INFO:teuthology.orchestra.run.smithi094.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2021-02-11T09:35:09.206 INFO:teuthology.orchestra.run.smithi094.stdout:Reading package lists... 2021-02-11T09:35:09.254 DEBUG:teuthology.orchestra.run.smithi094:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2021-02-11T09:35:09.338 INFO:teuthology.orchestra.run.smithi094.stdout:Reading package lists... 2021-02-11T09:35:09.584 INFO:teuthology.orchestra.run.smithi094.stdout:Building dependency tree... 2021-02-11T09:35:09.584 INFO:teuthology.orchestra.run.smithi094.stdout:Reading state information... 2021-02-11T09:35:09.671 INFO:teuthology.orchestra.run.smithi094.stderr:E: Unable to locate package linux-image-current-generic 2021-02-11T09:35:09.672 DEBUG:teuthology.orchestra.run:got remote process result: 100 2021-02-11T09:35:09.672 DEBUG:teuthology.orchestra.run.smithi094:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2021-02-11T09:35:09.753 INFO:teuthology.orchestra.run.smithi094.stdout:Reading package lists... 2021-02-11T09:35:10.016 INFO:teuthology.orchestra.run.smithi094.stdout:Building dependency tree... 2021-02-11T09:35:10.016 INFO:teuthology.orchestra.run.smithi094.stdout:Reading state information... 2021-02-11T09:35:10.228 INFO:teuthology.orchestra.run.smithi094.stdout:linux-image-generic is already the newest version (4.15.0.135.122). 2021-02-11T09:35:10.229 INFO:teuthology.orchestra.run.smithi094.stdout:0 upgraded, 0 newly installed, 0 to remove and 23 not upgraded. 2021-02-11T09:35:10.231 DEBUG:teuthology.orchestra.run.smithi094:> dpkg -s linux-image-generic 2021-02-11T09:35:10.252 INFO:teuthology.orchestra.run.smithi094.stdout:Package: linux-image-generic 2021-02-11T09:35:10.252 INFO:teuthology.orchestra.run.smithi094.stdout:Status: install ok installed 2021-02-11T09:35:10.252 INFO:teuthology.orchestra.run.smithi094.stdout:Priority: optional 2021-02-11T09:35:10.252 INFO:teuthology.orchestra.run.smithi094.stdout:Section: kernel 2021-02-11T09:35:10.252 INFO:teuthology.orchestra.run.smithi094.stdout:Installed-Size: 17 2021-02-11T09:35:10.252 INFO:teuthology.orchestra.run.smithi094.stdout:Maintainer: Ubuntu Kernel Team 2021-02-11T09:35:10.252 INFO:teuthology.orchestra.run.smithi094.stdout:Architecture: amd64 2021-02-11T09:35:10.252 INFO:teuthology.orchestra.run.smithi094.stdout:Source: linux-meta 2021-02-11T09:35:10.252 INFO:teuthology.orchestra.run.smithi094.stdout:Version: 4.15.0.135.122 2021-02-11T09:35:10.252 INFO:teuthology.orchestra.run.smithi094.stdout:Provides: wireguard-modules (= 1.0.20201112-1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.10) 2021-02-11T09:35:10.252 INFO:teuthology.orchestra.run.smithi094.stdout:Depends: linux-image-4.15.0-135-generic, linux-modules-extra-4.15.0-135-generic, linux-firmware, intel-microcode, amd64-microcode 2021-02-11T09:35:10.252 INFO:teuthology.orchestra.run.smithi094.stdout:Recommends: thermald 2021-02-11T09:35:10.252 INFO:teuthology.orchestra.run.smithi094.stdout:Description: Generic Linux kernel image 2021-02-11T09:35:10.253 INFO:teuthology.orchestra.run.smithi094.stdout: This package will always depend on the latest generic kernel image 2021-02-11T09:35:10.253 INFO:teuthology.orchestra.run.smithi094.stdout: available. 2021-02-11T09:35:10.253 DEBUG:teuthology.orchestra.run.smithi094:> mktemp 2021-02-11T09:35:10.300 INFO:teuthology.orchestra.run.smithi094.stdout:/tmp/tmp.QjBbaHq7YO 2021-02-11T09:35:10.301 DEBUG:teuthology.orchestra.run.smithi094:> sudo cp /boot/grub/grub.cfg /tmp/tmp.QjBbaHq7YO 2021-02-11T09:35:10.355 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 0666 /tmp/tmp.QjBbaHq7YO 2021-02-11T09:35:10.467 DEBUG:teuthology.orchestra.remote:smithi094:/tmp/tmp.QjBbaHq7YO is 8KB 2021-02-11T09:35:10.478 DEBUG:teuthology.orchestra.run.smithi094:> rm -fr /tmp/tmp.QjBbaHq7YO 2021-02-11T09:35:10.483 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2021-02-11T09:35:10.537 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2021-02-11T09:35:10.537 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/etc/grub.d/01_ceph_kernel 2021-02-11T09:35:10.537 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2021-02-11T09:35:10.605 INFO:teuthology.task.kernel:Distro Kernel Version: 4.15.0-135-generic 2021-02-11T09:35:10.605 DEBUG:teuthology.orchestra.run.smithi094:> sudo update-grub 2021-02-11T09:35:11.968 INFO:teuthology.orchestra.run.smithi094.stderr:Sourcing file `/etc/default/grub' 2021-02-11T09:35:14.533 INFO:teuthology.orchestra.run.smithi094.stderr:Generating grub configuration file ... 2021-02-11T09:35:14.534 INFO:teuthology.orchestra.run.smithi094.stderr:Found linux image: /boot/vmlinuz-4.15.0-135-generic 2021-02-11T09:35:14.534 INFO:teuthology.orchestra.run.smithi094.stderr:Found initrd image: /boot/initrd.img-4.15.0-135-generic 2021-02-11T09:35:14.534 INFO:teuthology.orchestra.run.smithi094.stderr:Found linux image: /boot/vmlinuz-4.15.0-134-generic 2021-02-11T09:35:14.534 INFO:teuthology.orchestra.run.smithi094.stderr:Found initrd image: /boot/initrd.img-4.15.0-134-generic 2021-02-11T09:35:14.534 INFO:teuthology.orchestra.run.smithi094.stderr:done 2021-02-11T09:35:14.534 DEBUG:teuthology.orchestra.run.smithi094:> sudo shutdown -r now 2021-02-11T09:35:14.537 INFO:teuthology.task.kernel:Installing distro kernel on c2.mon.a... 2021-02-11T09:35:14.537 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi098.front.sepia.ceph.com, path=None, version=distro) 2021-02-11T09:35:14.537 DEBUG:teuthology.orchestra.run.smithi098:> sudo apt-get clean 2021-02-11T09:35:14.578 DEBUG:teuthology.orchestra.run.smithi098:> sudo apt-get update 2021-02-11T09:35:14.749 INFO:teuthology.orchestra.run.smithi098.stdout:Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease 2021-02-11T09:35:14.852 INFO:teuthology.orchestra.run.smithi098.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2021-02-11T09:35:14.930 INFO:teuthology.orchestra.run.smithi098.stdout:Hit:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2021-02-11T09:35:15.011 INFO:teuthology.orchestra.run.smithi098.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2021-02-11T09:35:16.601 INFO:teuthology.orchestra.run.smithi098.stdout:Reading package lists... 2021-02-11T09:35:16.647 DEBUG:teuthology.orchestra.run.smithi098:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2021-02-11T09:35:16.718 INFO:teuthology.orchestra.run.smithi098.stdout:Reading package lists... 2021-02-11T09:35:16.953 INFO:teuthology.orchestra.run.smithi098.stdout:Building dependency tree... 2021-02-11T09:35:16.954 INFO:teuthology.orchestra.run.smithi098.stdout:Reading state information... 2021-02-11T09:35:17.035 INFO:teuthology.orchestra.run.smithi098.stderr:E: Unable to locate package linux-image-current-generic 2021-02-11T09:35:17.036 DEBUG:teuthology.orchestra.run:got remote process result: 100 2021-02-11T09:35:17.036 DEBUG:teuthology.orchestra.run.smithi098:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2021-02-11T09:35:17.159 INFO:teuthology.orchestra.run.smithi098.stdout:Reading package lists... 2021-02-11T09:35:17.417 INFO:teuthology.orchestra.run.smithi098.stdout:Building dependency tree... 2021-02-11T09:35:17.417 INFO:teuthology.orchestra.run.smithi098.stdout:Reading state information... 2021-02-11T09:35:17.614 INFO:teuthology.orchestra.run.smithi098.stdout:linux-image-generic is already the newest version (4.15.0.135.122). 2021-02-11T09:35:17.614 INFO:teuthology.orchestra.run.smithi098.stdout:0 upgraded, 0 newly installed, 0 to remove and 23 not upgraded. 2021-02-11T09:35:17.616 DEBUG:teuthology.orchestra.run.smithi098:> dpkg -s linux-image-generic 2021-02-11T09:35:17.639 INFO:teuthology.orchestra.run.smithi098.stdout:Package: linux-image-generic 2021-02-11T09:35:17.639 INFO:teuthology.orchestra.run.smithi098.stdout:Status: install ok installed 2021-02-11T09:35:17.640 INFO:teuthology.orchestra.run.smithi098.stdout:Priority: optional 2021-02-11T09:35:17.640 INFO:teuthology.orchestra.run.smithi098.stdout:Section: kernel 2021-02-11T09:35:17.640 INFO:teuthology.orchestra.run.smithi098.stdout:Installed-Size: 17 2021-02-11T09:35:17.640 INFO:teuthology.orchestra.run.smithi098.stdout:Maintainer: Ubuntu Kernel Team 2021-02-11T09:35:17.640 INFO:teuthology.orchestra.run.smithi098.stdout:Architecture: amd64 2021-02-11T09:35:17.640 INFO:teuthology.orchestra.run.smithi098.stdout:Source: linux-meta 2021-02-11T09:35:17.640 INFO:teuthology.orchestra.run.smithi098.stdout:Version: 4.15.0.135.122 2021-02-11T09:35:17.640 INFO:teuthology.orchestra.run.smithi098.stdout:Provides: wireguard-modules (= 1.0.20201112-1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.10) 2021-02-11T09:35:17.640 INFO:teuthology.orchestra.run.smithi098.stdout:Depends: linux-image-4.15.0-135-generic, linux-modules-extra-4.15.0-135-generic, linux-firmware, intel-microcode, amd64-microcode 2021-02-11T09:35:17.640 INFO:teuthology.orchestra.run.smithi098.stdout:Recommends: thermald 2021-02-11T09:35:17.640 INFO:teuthology.orchestra.run.smithi098.stdout:Description: Generic Linux kernel image 2021-02-11T09:35:17.640 INFO:teuthology.orchestra.run.smithi098.stdout: This package will always depend on the latest generic kernel image 2021-02-11T09:35:17.641 INFO:teuthology.orchestra.run.smithi098.stdout: available. 2021-02-11T09:35:17.641 DEBUG:teuthology.orchestra.run.smithi098:> mktemp 2021-02-11T09:35:17.688 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/tmp.3TmKLUaPE7 2021-02-11T09:35:17.689 DEBUG:teuthology.orchestra.run.smithi098:> sudo cp /boot/grub/grub.cfg /tmp/tmp.3TmKLUaPE7 2021-02-11T09:35:17.758 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0666 /tmp/tmp.3TmKLUaPE7 2021-02-11T09:35:17.874 DEBUG:teuthology.orchestra.remote:smithi098:/tmp/tmp.3TmKLUaPE7 is 8KB 2021-02-11T09:35:17.925 DEBUG:teuthology.orchestra.run.smithi098:> rm -fr /tmp/tmp.3TmKLUaPE7 2021-02-11T09:35:17.931 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2021-02-11T09:35:17.989 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2021-02-11T09:35:17.989 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/grub.d/01_ceph_kernel 2021-02-11T09:35:17.989 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2021-02-11T09:35:18.060 INFO:teuthology.task.kernel:Distro Kernel Version: 4.15.0-135-generic 2021-02-11T09:35:18.061 DEBUG:teuthology.orchestra.run.smithi098:> sudo update-grub 2021-02-11T09:35:19.458 INFO:teuthology.orchestra.run.smithi098.stderr:Sourcing file `/etc/default/grub' 2021-02-11T09:35:19.480 INFO:teuthology.orchestra.run.smithi098.stderr:Generating grub configuration file ... 2021-02-11T09:35:19.724 INFO:teuthology.orchestra.run.smithi098.stderr:Found linux image: /boot/vmlinuz-4.15.0-135-generic 2021-02-11T09:35:19.737 INFO:teuthology.orchestra.run.smithi098.stderr:Found initrd image: /boot/initrd.img-4.15.0-135-generic 2021-02-11T09:35:20.035 INFO:teuthology.orchestra.run.smithi098.stderr:Found linux image: /boot/vmlinuz-4.15.0-134-generic 2021-02-11T09:35:20.047 INFO:teuthology.orchestra.run.smithi098.stderr:Found initrd image: /boot/initrd.img-4.15.0-134-generic 2021-02-11T09:35:21.157 INFO:teuthology.orchestra.run.smithi098.stderr:done 2021-02-11T09:35:21.159 DEBUG:teuthology.orchestra.run.smithi098:> sudo shutdown -r now 2021-02-11T09:35:51.234 INFO:teuthology.misc:Re-opening connections... 2021-02-11T09:35:51.235 INFO:teuthology.misc:trying to connect to ubuntu@smithi094.front.sepia.ceph.com 2021-02-11T09:35:51.235 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-11T09:35:51.236 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-02-11T09:36:09.666 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.94 2021-02-11T09:36:09.667 INFO:teuthology.misc:trying to connect to ubuntu@smithi098.front.sepia.ceph.com 2021-02-11T09:36:09.667 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-11T09:36:09.667 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-02-11T09:36:28.102 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.98 2021-02-11T09:36:28.103 DEBUG:teuthology.misc:waited 36.8678457736969 2021-02-11T09:36:29.103 INFO:teuthology.misc:trying to connect to ubuntu@smithi094.front.sepia.ceph.com 2021-02-11T09:36:29.103 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-11T09:36:29.104 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-02-11T09:36:47.554 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.94 2021-02-11T09:36:47.555 INFO:teuthology.misc:trying to connect to ubuntu@smithi098.front.sepia.ceph.com 2021-02-11T09:36:47.555 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-11T09:36:47.555 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-02-11T09:37:05.986 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.98 2021-02-11T09:37:05.987 DEBUG:teuthology.misc:waited 74.7521059513092 2021-02-11T09:37:06.987 INFO:teuthology.misc:trying to connect to ubuntu@smithi094.front.sepia.ceph.com 2021-02-11T09:37:06.987 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-11T09:37:06.988 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-02-11T09:37:07.069 DEBUG:teuthology.orchestra.run.smithi094:> true 2021-02-11T09:37:07.674 DEBUG:teuthology.misc:waited 76.43980956077576 2021-02-11T09:37:08.675 INFO:teuthology.misc:trying to connect to ubuntu@smithi098.front.sepia.ceph.com 2021-02-11T09:37:08.676 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-11T09:37:08.676 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-02-11T09:37:08.754 DEBUG:teuthology.orchestra.run.smithi098:> true 2021-02-11T09:37:09.114 DEBUG:teuthology.misc:waited 77.87954902648926 2021-02-11T09:37:10.115 INFO:teuthology.task.kernel:Checking client c1.mon.a for new kernel version... 2021-02-11T09:37:10.115 INFO:teuthology.task.kernel:Checking kernel version of c1.mon.a, want "4.15.0-135-generic"... 2021-02-11T09:37:10.115 DEBUG:teuthology.orchestra.run.smithi094:> uname -r 2021-02-11T09:37:10.121 INFO:teuthology.orchestra.run.smithi094.stdout:4.15.0-135-generic 2021-02-11T09:37:10.122 DEBUG:teuthology.task.kernel:current kernel version is 4.15.0-135-generic vs 4.15.0-135-generic 2021-02-11T09:37:10.122 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2021-02-11T09:37:10.122 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2021-02-11T09:37:10.122 INFO:teuthology.task.kernel:Checking client c2.mon.a for new kernel version... 2021-02-11T09:37:10.122 INFO:teuthology.task.kernel:Checking kernel version of c2.mon.a, want "4.15.0-135-generic"... 2021-02-11T09:37:10.122 DEBUG:teuthology.orchestra.run.smithi098:> uname -r 2021-02-11T09:37:10.128 INFO:teuthology.orchestra.run.smithi098.stdout:4.15.0-135-generic 2021-02-11T09:37:10.128 DEBUG:teuthology.task.kernel:current kernel version is 4.15.0-135-generic vs 4.15.0-135-generic 2021-02-11T09:37:10.128 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2021-02-11T09:37:10.128 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2021-02-11T09:37:11.129 INFO:teuthology.task.kernel:Enabling kdb on c1.mon.a... 2021-02-11T09:37:11.129 DEBUG:teuthology.orchestra.run.smithi094:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-02-11T09:37:11.223 INFO:teuthology.orchestra.run.smithi094.stdout:ttyS1 2021-02-11T09:37:11.259 INFO:teuthology.task.kernel:Enabling kdb on c2.mon.a... 2021-02-11T09:37:11.260 DEBUG:teuthology.orchestra.run.smithi098:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-02-11T09:37:11.290 INFO:teuthology.orchestra.run.smithi098.stdout:ttyS1 2021-02-11T09:37:11.313 INFO:teuthology.run_tasks:Running task internal.base... 2021-02-11T09:37:11.321 INFO:teuthology.task.internal:Creating test directory... 2021-02-11T09:37:11.321 DEBUG:teuthology.orchestra.run.smithi094:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-02-11T09:37:11.324 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-02-11T09:37:11.361 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2021-02-11T09:37:11.555 INFO:teuthology.run_tasks:Running task internal.archive... 2021-02-11T09:37:18.358 INFO:teuthology.task.internal:Creating archive directory... 2021-02-11T09:37:18.359 DEBUG:teuthology.orchestra.run.smithi094:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-02-11T09:37:18.361 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-02-11T09:37:18.384 INFO:teuthology.run_tasks:Running task internal.coredump... 2021-02-11T09:37:27.096 INFO:teuthology.task.internal:Enabling coredump saving... 2021-02-11T09:37:27.096 DEBUG:teuthology.orchestra.run.smithi094:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-02-11T09:37:27.099 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-02-11T09:37:27.119 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-02-11T09:37:27.128 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-02-11T09:37:27.227 INFO:teuthology.orchestra.run.smithi094.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-02-11T09:37:27.237 INFO:teuthology.orchestra.run.smithi094.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-02-11T09:37:27.239 INFO:teuthology.run_tasks:Running task internal.sudo... 2021-02-11T09:37:27.246 INFO:teuthology.task.internal:Configuring sudo... 2021-02-11T09:37:27.246 DEBUG:teuthology.orchestra.run.smithi094:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-02-11T09:37:27.248 DEBUG:teuthology.orchestra.run.smithi098:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-02-11T09:37:27.263 INFO:teuthology.run_tasks:Running task internal.syslog... 2021-02-11T09:37:27.373 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2021-02-11T09:37:27.373 DEBUG:teuthology.orchestra.run.smithi094:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-02-11T09:37:27.375 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-02-11T09:37:27.382 DEBUG:teuthology.orchestra.run.smithi094:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-02-11T09:37:27.427 DEBUG:teuthology.orchestra.run.smithi094:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-02-11T09:37:27.475 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2021-02-11T09:37:27.475 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-02-11T09:37:27.547 DEBUG:teuthology.orchestra.run.smithi098:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-02-11T09:37:27.553 DEBUG:teuthology.orchestra.run.smithi098:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-02-11T09:37:27.600 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2021-02-11T09:37:27.600 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-02-11T09:37:27.675 DEBUG:teuthology.orchestra.run.smithi094:> sudo service rsyslog restart 2021-02-11T09:37:27.678 DEBUG:teuthology.orchestra.run.smithi098:> sudo service rsyslog restart 2021-02-11T09:37:27.780 INFO:teuthology.run_tasks:Running task internal.timer... 2021-02-11T09:37:27.787 INFO:teuthology.task.internal:Starting timer... 2021-02-11T09:37:27.788 INFO:teuthology.run_tasks:Running task pcp... 2021-02-11T09:37:27.903 INFO:teuthology.run_tasks:Running task selinux... 2021-02-11T09:37:27.940 DEBUG:teuthology.task.selinux:Excluding smithi094: OS 'ubuntu' does not support SELinux 2021-02-11T09:37:27.941 DEBUG:teuthology.task.selinux:Excluding smithi098: OS 'ubuntu' does not support SELinux 2021-02-11T09:37:27.941 DEBUG:teuthology.task.selinux:Getting current SELinux state 2021-02-11T09:37:27.941 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2021-02-11T09:37:27.941 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2021-02-11T09:37:27.941 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2021-02-11T09:37:28.380 DEBUG:teuthology.repo_utils:Setting repo remote to git://git.ceph.com/git/ceph-cm-ansible.git 2021-02-11T09:37:28.403 INFO:teuthology.repo_utils:Fetching master from origin 2021-02-11T09:37:28.640 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2021-02-11T09:37:28.897 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}] 2021-02-11T09:37:28.911 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi094.front.sepia.ceph.com,smithi098.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2021-02-11T09:37:29.652 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2021-02-11T09:37:31.457 INFO:teuthology.task.ansible.out:[WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2021-02-11T09:37:34.206 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2021-02-11T09:37:34.262 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:37:34 +0000 (0:00:00.183) 0:00:00.183 ***** 2021-02-11T09:37:34.283 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:37:34 +0000 (0:00:00.020) 0:00:00.203 ***** 2021-02-11T09:37:36.053 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi094.front.sepia.ceph.com] 2021-02-11T09:37:36.097 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] 2021-02-11T09:37:36.133 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:37:36 +0000 (0:00:01.850) 0:00:02.053 ***** 2021-02-11T09:37:36.195 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:37:36 +0000 (0:00:00.062) 0:00:02.116 ***** 2021-02-11T09:37:36.367 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2021-02-11T09:37:36.369 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2021-02-11T09:37:36.400 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:37:36 +0000 (0:00:00.204) 0:00:02.321 ***** 2021-02-11T09:37:36.406 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2021-02-11T09:37:36.441 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:37:36 +0000 (0:00:00.040) 0:00:02.362 ***** 2021-02-11T09:37:36.798 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2021-02-11T09:37:36.800 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => {"append": false, "changed": true, "comment": "cm,,,", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2021-02-11T09:37:36.833 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:37:36 +0000 (0:00:00.392) 0:00:02.754 ***** 2021-02-11T09:37:36.840 INFO:teuthology.task.ansible.out:changed: [smithi098.front.sepia.ceph.com] => {"append": false, "changed": true, "comment": "cm,,,", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2021-02-11T09:37:36.875 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:37:36 +0000 (0:00:00.041) 0:00:02.795 ***** 2021-02-11T09:37:37.138 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2021-02-11T09:37:37.140 INFO:teuthology.task.ansible.out:changed: [smithi098.front.sepia.ceph.com] => {"changed": true, "cmd": ["passwd", "-d", "cm"], "delta": "0:00:00.068060", "end": "2021-02-11 09:37:37.432453", "rc": 0, "start": "2021-02-11 09:37:37.364393", "stderr": "", "stderr_lines": [], "stdout": "passwd: password expiry information changed.", "stdout_lines": ["passwd: password expiry information changed."]} 2021-02-11T09:37:37.171 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:37:37 +0000 (0:00:00.296) 0:00:03.092 ***** 2021-02-11T09:37:37.176 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => {"changed": true, "cmd": ["passwd", "-d", "cm"], "delta": "0:00:00.091537", "end": "2021-02-11 09:37:37.135759", "rc": 0, "start": "2021-02-11 09:37:37.044222", "stderr": "", "stderr_lines": [], "stdout": "passwd: password expiry information changed.", "stdout_lines": ["passwd: password expiry information changed."]} 2021-02-11T09:37:37.213 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:37:37 +0000 (0:00:00.041) 0:00:03.133 ***** 2021-02-11T09:37:37.404 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** 2021-02-11T09:37:37.405 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} ok: [smithi094.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-11T09:37:37.439 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:37:37 +0000 (0:00:00.226) 0:00:03.360 ***** 2021-02-11T09:37:37.470 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:37:37 +0000 (0:00:00.030) 0:00:03.390 ***** 2021-02-11T09:37:38.181 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2021-02-11T09:37:38.183 INFO:teuthology.task.ansible.out:changed: [smithi098.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1613036257.4827433-175752020684917/source", "state": "file", "uid": 0} changed: [smithi094.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1613036257.4521143-68944705621490/source", "state": "file", "uid": 0} 2021-02-11T09:37:38.219 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:37:38 +0000 (0:00:00.749) 0:00:04.139 ***** 2021-02-11T09:37:38.253 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:37:38 +0000 (0:00:00.033) 0:00:04.173 ***** 2021-02-11T09:37:38.506 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2021-02-11T09:37:38.507 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2021-02-11T09:37:38.508 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} ok: [smithi098.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail 2021-02-11T09:37:38.510 INFO:teuthology.task.ansible.out:.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@ 2021-02-11T09:37:38.511 INFO:teuthology.task.ansible.out:aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2021-02-11T09:37:38.638 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2021-02-11T09:37:38.645 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-02-11T09:37:38.788 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:37:38 +0000 (0:00:00.535) 0:00:04.708 ***** 2021-02-11T09:37:38.822 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:37:38 +0000 (0:00:00.033) 0:00:04.742 ***** 2021-02-11T09:37:39.505 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi094.front.sepia.ceph.com] 2021-02-11T09:37:39.595 INFO:teuthology.task.ansible.out:[WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. 2021-02-11T09:37:39.595 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:37:39 +0000 (0:00:00.773) 0:00:05.516 ***** 2021-02-11T09:37:40.410 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2021-02-11T09:37:42.342 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.473783", "end": "2021-02-11 09:37:40.383174", "rc": 0, "start": "2021-02-11 09:37:39.909391", "stderr": "", "stderr_lines": [], "stdout": "df8946f08e870e2d2565b79099289ca48bd825f3", "stdout_lines": ["df8946f08e870e2d2565b79099289ca48bd825f3"]} Thursday 11 February 2021 09:37:40 +0000 (0:00:00.851) 0:00:06.367 ***** TASK [Gathering Facts] ********************************************************* ok: [smithi098.front.sepia.ceph.com] Thursday 11 February 2021 09:37:40 +0000 (0:00:00.125) 0:00:06.492 ***** TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** changed: [smithi094.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.012255", "end": "2021-02-11 09:37:40.563160", "failed_when_result": false, "rc": 0, "start": "2021-02-11 09:37:40.550905", "stderr": "", "stderr_lines": [], "stdout": "0caff10a478f2674721da1965013b225f3ea5800", "stdout_lines": ["0caff10a478f2674721da1965013b225f3ea5800"]} Thursday 11 February 2021 09:37:40 +0000 (0:00:00.054) 0:00:06.546 ***** TASK [users : Determine if we can skip users and pubkeys updates] ************** skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 09:37:40 +0000 (0:00:00.061) 0:00:06.608 ***** TASK [users : Merge extra_admin_users into managed_admin_users] **************** skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 09:37:40 +0000 (0:00:00.060) 0:00:06.668 ***** TASK [users : Remove managed_admin_users from managed_users] ******************* skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional resul 2021-02-11T09:37:42.344 INFO:teuthology.task.ansible.out:t was False"} Thursday 11 February 2021 09:37:40 +0000 (0:00:00.060) 0:00:06.729 ***** TASK [users : Filter the managed_users list] *********************************** skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 09:37:40 +0000 (0:00:00.059) 0:00:06.788 ***** TASK [users : Filter the managed_admin_users list] ***************************** skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 09:37:40 +0000 (0:00:00.058) 0:00:06.847 ***** TASK [users : Check keys_repo HEAD sha1] *************************************** changed: [smithi098.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.157315", "end": "2021-02-11 09:37:40.907000", "rc": 0, "start": "2021-02-11 09:37:40.749685", "stderr": "", "stderr_lines": [], "stdout": "df8946f08e870e2d2565b79099289ca48bd825f3", "stdout_lines": ["df8946f08e870e2d2565b79099289ca48bd825f3"]} Thursday 11 February 2021 09:37:40 +0000 (0:00:00.042) 0:00:06.890 ***** TASK [users : Set UID_MIN to 1001] ********************************************* ok: [smithi094.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} Thursday 11 February 2021 09:37:41 +0000 (0:00:00.113) 0:00:07.003 ***** TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** changed: [smithi098.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.016481", "end": "2021-02-11 09:37:41.411050", "failed_when_result": false, "rc": 0, "start": "2021-02-11 09:37:41.394569", "stderr": "", "stderr_lines": [], "stdout": "0caff10a478f2674721da1965013b225f3ea5800", "stdout_lines": ["0caff10a478f2674721da1965013b225f3ea5800"]} Thursday 11 February 2021 09:37:41 2021-02-11T09:37:42.345 INFO:teuthology.task.ansible.out:+0000 (0:00:00.074) 0:00:07.077 ***** TASK [users : Determine if we can skip users and pubkeys updates] ************** skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 09:37:41 +0000 (0:00:00.059) 0:00:07.137 ***** TASK [users : Merge extra_admin_users into managed_admin_users] **************** skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 09:37:41 +0000 (0:00:00.059) 0:00:07.197 ***** TASK [users : Create all admin users with sudo access.] ************************ ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} TASK [users : Remove managed_admin_users from managed_users] ******************* skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 09:37:41 +0000 (0:00:00.060) 0:00:07.257 ***** TASK [users : Filter the managed_users list] *********************************** skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 09:37:41 +0000 (0:00:00.058) 0:00:07.316 ***** TASK [users : Create all admin users with sudo access.] ************************ ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1Ce 2021-02-11T09:37:42.346 INFO:teuthology.task.ansible.out:ZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} TASK [users : Filter the managed_admin_users list] ***************************** skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 09:37:41 +0000 (0:00:00.060) 0:00:07.376 ***** TASK [users : Create all admin users with sudo access.] ************************ ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} TASK [users : Set UID_MIN to 1001] ********************************************* ok: [smithi098.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} Thursday 11 February 2021 09:37:41 +0000 (0:00:00.153) 0:00:07.529 ***** TASK [users : Create all admin users with sudo access.] ************************ ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 2021-02-11T09:37:42.348 INFO:teuthology.task.ansible.out:'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631 2021-02-11T09:37:42.349 INFO:teuthology.task.ansible.out:754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba 2021-02-11T09:37:42.350 INFO:teuthology.task.ansible.out:.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "hom 2021-02-11T09:37:42.352 INFO:teuthology.task.ansible.out:e": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b 2021-02-11T09:37:42.352 INFO:teuthology.task.ansible.out:9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-02-11T09:37:42.353 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-02-11T09:37:42.354 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-02-11T09:37:42.376 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-02-11T09:37:42.419 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-02-11T09:37:42.534 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-02-11T09:37:42.545 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-02-11T09:37:42.649 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-02-11T09:37:42.691 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2021-02-11T09:37:42.785 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-02-11T09:37:42.807 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-02-11T09:37:42.901 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-02-11T09:37:42.932 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-02-11T09:37:43.027 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-02-11T09:37:43.059 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-02-11T09:37:43.142 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-02-11T09:37:43.185 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-02-11T09:37:43.268 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-02-11T09:37:43.311 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-02-11T09:37:43.394 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2021-02-11T09:37:43.436 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-02-11T09:37:43.520 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-02-11T09:37:43.562 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-02-11T09:37:43.645 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-02-11T09:37:43.711 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-02-11T09:37:43.763 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2021-02-11T09:37:43.847 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-02-11T09:37:43.889 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-02-11T09:37:43.973 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2021-02-11T09:37:44.015 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-02-11T09:37:44.111 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-02-11T09:37:44.143 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-02-11T09:37:44.237 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-02-11T09:37:44.268 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-02-11T09:37:44.373 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2021-02-11T09:37:44.395 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-02-11T09:37:44.520 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-02-11T09:37:44.521 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-02-11T09:37:44.646 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-02-11T09:37:44.647 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-02-11T09:37:44.772 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-02-11T09:37:44.793 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-02-11T09:37:44.887 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-02-11T09:37:44.929 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-02-11T09:37:45.012 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-02-11T09:37:45.065 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-02-11T09:37:45.137 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1133} 2021-02-11T09:37:45.211 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-02-11T09:37:45.263 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-02-11T09:37:45.347 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-02-11T09:37:45.389 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-02-11T09:37:45.472 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-02-11T09:37:45.515 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-02-11T09:37:45.608 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-02-11T09:37:45.640 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1137} 2021-02-11T09:37:45.744 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-02-11T09:37:45.755 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-02-11T09:37:45.880 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1133} 2021-02-11T09:37:45.881 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-02-11T09:37:46.006 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-02-11T09:37:46.017 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-02-11T09:37:46.122 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-02-11T09:37:46.164 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-02-11T09:37:46.248 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-02-11T09:37:46.300 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-02-11T09:37:46.373 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-02-11T09:37:46.436 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1137} 2021-02-11T09:37:46.499 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-02-11T09:37:46.573 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-02-11T09:37:46.625 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-02-11T09:37:46.709 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-02-11T09:37:46.751 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-02-11T09:37:46.845 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-02-11T09:37:46.867 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-02-11T09:37:46.971 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-02-11T09:37:46.992 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-02-11T09:37:47.118 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-02-11T09:37:47.119 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-02-11T09:37:47.245 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-02-11T09:37:47.246 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-02-11T09:37:47.360 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-02-11T09:37:47.371 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-02-11T09:37:47.476 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-02-11T09:37:47.507 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-02-11T09:37:47.601 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-02-11T09:37:47.633 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-02-11T09:37:47.727 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-02-11T09:37:47.770 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-02-11T09:37:47.843 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-02-11T09:37:47.906 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-02-11T09:37:47.969 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-02-11T09:37:48.043 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-02-11T09:37:48.095 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-02-11T09:37:48.179 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-02-11T09:37:48.211 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-02-11T09:37:48.316 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-02-11T09:37:48.337 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-02-11T09:37:48.463 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-02-11T09:37:48.464 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-02-11T09:37:48.582 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-02-11T09:37:48.583 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-02-11T09:37:48.708 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-02-11T09:37:48.709 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-02-11T09:37:48.834 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-02-11T09:37:48.835 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-02-11T09:37:48.949 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-02-11T09:37:48.960 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-02-11T09:37:49.085 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-02-11T09:37:49.097 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-02-11T09:37:49.211 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-02-11T09:37:49.243 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-02-11T09:37:49.338 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-02-11T09:37:49.369 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-02-11T09:37:49.464 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-02-11T09:37:49.506 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-02-11T09:37:49.579 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-02-11T09:37:49.642 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-02-11T09:37:49.706 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-02-11T09:37:49.779 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-02-11T09:37:49.821 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-02-11T09:37:49.905 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-02-11T09:37:49.947 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-02-11T09:37:50.042 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-02-11T09:37:50.073 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-02-11T09:37:50.188 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-02-11T09:37:50.189 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-02-11T09:37:50.317 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-02-11T09:37:50.328 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-02-11T09:37:50.432 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-02-11T09:37:50.474 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-02-11T09:37:50.558 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-02-11T09:37:50.592 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-02-11T09:37:50.686 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-02-11T09:37:50.718 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-02-11T09:37:50.801 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-02-11T09:37:50.854 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-02-11T09:37:50.927 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-02-11T09:37:50.990 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-02-11T09:37:51.053 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-02-11T09:37:51.137 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-02-11T09:37:51.179 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-02-11T09:37:51.263 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-02-11T09:37:51.295 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-02-11T09:37:51.389 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-02-11T09:37:51.421 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1183} 2021-02-11T09:37:51.526 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-02-11T09:37:51.538 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1184} 2021-02-11T09:37:51.663 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-02-11T09:37:51.665 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-02-11T09:37:51.791 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-02-11T09:37:51.791 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-02-11T09:37:51.906 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-02-11T09:37:51.928 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-02-11T09:37:52.032 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-02-11T09:37:52.074 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-02-11T09:37:52.158 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-02-11T09:37:52.200 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-02-11T09:37:52.274 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-02-11T09:37:52.327 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-02-11T09:37:52.400 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-02-11T09:37:52.453 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-02-11T09:37:52.517 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-02-11T09:37:52.583 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-02-11T09:37:52.646 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-02-11T09:37:52.708 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1183} 2021-02-11T09:37:52.761 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-02-11T09:37:52.835 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1184} 2021-02-11T09:37:52.887 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-02-11T09:37:52.981 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-02-11T09:37:53.013 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-02-11T09:37:53.117 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-02-11T09:37:53.128 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-02-11T09:37:53.254 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-02-11T09:37:53.255 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-02-11T09:37:53.380 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-02-11T09:37:53.391 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-02-11T09:37:53.506 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-02-11T09:37:53.528 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-02-11T09:37:53.622 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-02-11T09:37:53.654 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-02-11T09:37:53.748 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-02-11T09:37:53.791 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-02-11T09:37:53.875 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-02-11T09:37:53.928 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-02-11T09:37:54.001 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-02-11T09:37:54.064 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-02-11T09:37:54.116 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-02-11T09:37:54.200 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-02-11T09:37:54.242 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-02-11T09:37:54.336 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-02-11T09:37:54.368 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-02-11T09:37:54.471 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-02-11T09:37:54.493 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-02-11T09:37:54.599 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-02-11T09:37:54.610 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-02-11T09:37:54.735 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-02-11T09:37:54.736 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-02-11T09:37:54.850 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-02-11T09:37:54.861 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-02-11T09:37:54.975 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-02-11T09:37:54.996 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-02-11T09:37:55.100 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-02-11T09:37:55.132 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-02-11T09:37:55.227 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-02-11T09:37:55.269 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-02-11T09:37:55.352 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-02-11T09:37:55.405 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-02-11T09:37:55.469 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-02-11T09:37:55.553 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-02-11T09:37:55.616 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-02-11T09:37:55.699 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-02-11T09:37:55.741 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-02-11T09:37:55.846 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-02-11T09:37:55.867 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-02-11T09:37:55.982 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-02-11T09:37:55.983 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-02-11T09:37:56.108 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-02-11T09:37:56.119 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-02-11T09:37:56.245 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-02-11T09:37:56.256 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-02-11T09:37:56.361 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-02-11T09:37:56.392 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-02-11T09:37:56.486 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-02-11T09:37:56.529 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-02-11T09:37:56.615 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-02-11T09:37:56.667 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-02-11T09:37:56.730 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-02-11T09:37:56.814 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-02-11T09:37:56.856 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-02-11T09:37:56.950 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-02-11T09:37:56.981 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-02-11T09:37:57.075 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-02-11T09:37:57.107 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-02-11T09:37:57.212 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-02-11T09:37:57.233 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-02-11T09:37:57.338 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-02-11T09:37:57.359 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1230, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1230} 2021-02-11T09:37:57.464 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-02-11T09:37:57.485 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-02-11T09:37:57.590 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-02-11T09:37:57.601 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/hmunjulu", "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "move_home": false, "name": "hmunjulu", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-02-11T09:37:57.726 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-02-11T09:37:57.727 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-02-11T09:37:57.852 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1234, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1234} 2021-02-11T09:37:57.863 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-02-11T09:37:57.977 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-02-11T09:37:57.978 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "move_home": false, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-02-11T09:37:58.103 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1236, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "move_home": false, "name": "adking", "shell": "/bin/bash", "state": "present", "uid": 1236} 2021-02-11T09:37:58.115 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-02-11T09:37:58.230 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1237, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, "move_home": false, "name": "amathuri", "shell": "/bin/bash", "state": "present", "uid": 1237} 2021-02-11T09:37:58.262 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-02-11T09:37:58.387 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-02-11T09:37:58.501 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-02-11T09:37:58.626 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-02-11T09:37:58.761 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-02-11T09:37:58.907 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-02-11T09:37:59.041 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-02-11T09:37:59.157 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2021-02-11T09:37:59.238 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1230, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1230} 2021-02-11T09:37:59.276 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:37:59 +0000 (0:00:17.667) 0:00:25.197 ***** 2021-02-11T09:37:59.299 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2021-02-11T09:37:59.300 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2021-02-11T09:37:59.301 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-02-11T09:37:59.336 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:37:59 +0000 (0:00:00.059) 0:00:25.256 ***** 2021-02-11T09:37:59.384 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* 2021-02-11T09:37:59.386 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@fedora 2021-02-11T09:37:59.387 INFO:teuthology.task.ansible.out:rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265 2021-02-11T09:37:59.389 INFO:teuthology.task.ansible.out:c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancic 2021-02-11T09:37:59.390 INFO:teuthology.task.ansible.out:h", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1Wi 2021-02-11T09:37:59.392 INFO:teuthology.task.ansible.out:RenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdilla 2021-02-11T09:37:59.392 INFO:teuthology.task.ansible.out:man@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vum 2021-02-11T09:37:59.393 INFO:teuthology.task.ansible.out:rao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69 2021-02-11T09:37:59.394 INFO:teuthology.task.ansible.out:mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d8 2021-02-11T09:37:59.396 INFO:teuthology.task.ansible.out:41a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveirada 2021-02-11T09:37:59.397 INFO:teuthology.task.ansible.out:n@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOe 2021-02-11T09:37:59.398 INFO:teuthology.task.ansible.out:nVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "o 2021-02-11T09:37:59.398 INFO:teuthology.task.ansible.out:vpn": "rraja@p50 4NMUghtas9 2021-02-11T09:37:59.400 INFO:teuthology.task.ansible.out:gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": 2021-02-11T09:37:59.401 INFO:teuthology.task.ansible.out:"dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NI 2021-02-11T09:37:59.402 INFO:teuthology.task.ansible.out:Nj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}]}, "changed": false} 2021-02-11T09:37:59.438 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:37:59 +0000 (0:00:00.101) 0:00:25.358 ***** 2021-02-11T09:37:59.454 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2021-02-11T09:37:59.454 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/hmunjulu", "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "move_home": false, "name": "hmunjulu", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-02-11T09:37:59.579 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-02-11T09:37:59.705 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1234, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1234} 2021-02-11T09:37:59.841 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "move_home": false, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-02-11T09:37:59.987 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1236, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "move_home": false, "name": "adking", "shell": "/bin/bash", "state": "present", "uid": 1236} 2021-02-11T09:38:00.133 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1237, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, "move_home": false, "name": "amathuri", "shell": "/bin/bash", "state": "present", "uid": 1237} 2021-02-11T09:38:00.135 INFO:teuthology.task.ansible.out: 2021-02-11T09:38:00.517 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2021-02-11T09:38:00.518 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com -> localhost] => {"after": "df8946f08e870e2d2565b79099289ca48bd825f3", "attempts": 1, "before": "df8946f08e870e2d2565b79099289ca48bd825f3", "changed": false, "remote_url_changed": false} 2021-02-11T09:38:00.551 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:00 +0000 (0:00:01.113) 0:00:26.472 ***** 2021-02-11T09:38:00.743 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2021-02-11T09:38:00.745 INFO:teuthology.task.ansible.out:changed: [smithi098.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2021-02-11T09:38:00.851 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:00 +0000 (0:00:00.299) 0:00:26.772 ***** 2021-02-11T09:38:00.878 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2021-02-11T09:38:00.909 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:00 +0000 (0:00:00.058) 0:00:26.830 ***** 2021-02-11T09:38:00.946 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* 2021-02-11T09:38:00.948 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@fedora 2021-02-11T09:38:00.949 INFO:teuthology.task.ansible.out:rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265 2021-02-11T09:38:00.951 INFO:teuthology.task.ansible.out:c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancic 2021-02-11T09:38:00.952 INFO:teuthology.task.ansible.out:h", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1Wi 2021-02-11T09:38:00.953 INFO:teuthology.task.ansible.out:RenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdilla 2021-02-11T09:38:00.954 INFO:teuthology.task.ansible.out:man@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vum 2021-02-11T09:38:00.955 INFO:teuthology.task.ansible.out:rao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69 2021-02-11T09:38:00.956 INFO:teuthology.task.ansible.out:mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d8 2021-02-11T09:38:00.958 INFO:teuthology.task.ansible.out:41a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveirada 2021-02-11T09:38:00.959 INFO:teuthology.task.ansible.out:n@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOe 2021-02-11T09:38:00.960 INFO:teuthology.task.ansible.out:nVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "o 2021-02-11T09:38:00.960 INFO:teuthology.task.ansible.out:vpn": "rraja@p50 4NMUghtas9 2021-02-11T09:38:00.962 INFO:teuthology.task.ansible.out:gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": 2021-02-11T09:38:00.963 INFO:teuthology.task.ansible.out:"dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NI 2021-02-11T09:38:00.964 INFO:teuthology.task.ansible.out:Nj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}]}, "changed": false} TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh 2021-02-11T09:38:00.966 INFO:teuthology.task.ansible.out:-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-02-11T09:38:01.000 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:00 +0000 (0:00:00.090) 0:00:26.920 ***** 2021-02-11T09:38:01.088 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-02-11T09:38:01.215 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-02-11T09:38:01.350 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2021-02-11T09:38:01.351 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2021-02-11T09:38:01.464 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-02-11T09:38:01.621 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2021-02-11T09:38:01.622 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2021-02-11T09:38:01.745 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-02-11T09:38:01.860 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-02-11T09:38:01.975 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-02-11T09:38:01.986 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:02.110 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2021-02-11T09:38:02.112 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2021-02-11T09:38:02.226 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2021-02-11T09:38:02.227 INFO:teuthology.task.ansible.out:son", "validate_certs": true} 2021-02-11T09:38:02.237 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:02.238 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:02.362 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-02-11T09:38:02.486 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2021-02-11T09:38:02.487 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-02-11T09:38:02.601 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-02-11T09:38:02.726 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "davidz", "validate_certs": true} 2021-02-11T09:38:02.840 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-02-11T09:38:02.965 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-02-11T09:38:03.079 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2021-02-11T09:38:03.203 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2021-02-11T09:38:03.204 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2021-02-11T09:38:03.276 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2021-02-11T09:38:03.278 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com -> localhost] => {"after": "df8946f08e870e2d2565b79099289ca48bd825f3", "attempts": 1, "before": "df8946f08e870e2d2565b79099289ca48bd825f3", "changed": false, "remote_url_changed": false} 2021-02-11T09:38:03.309 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:03 +0000 (0:00:02.309) 0:00:29.229 ***** 2021-02-11T09:38:03.324 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-11T09:38:03.325 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2021-02-11T09:38:03.326 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2021-02-11T09:38:03.439 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-02-11T09:38:03.492 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-02-11T09:38:03.566 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-02-11T09:38:03.629 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-02-11T09:38:03.681 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46 2021-02-11T09:38:03.682 INFO:teuthology.task.ansible.out:TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-02-11T09:38:03.775 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-02-11T09:38:03.796 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-02-11T09:38:03.911 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2021-02-11T09:38:03.912 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2021-02-11T09:38:03.914 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-02-11T09:38:04.037 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-02-11T09:38:04.038 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-02-11T09:38:04.153 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-02-11T09:38:04.185 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2021-02-11T09:38:04.186 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2021-02-11T09:38:04.290 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-02-11T09:38:04.311 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-02-11T09:38:04.426 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2021-02-11T09:38:04.427 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LT 2021-02-11T09:38:04.429 INFO:teuthology.task.ansible.out:MoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-02-11T09:38:04.551 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2021-02-11T09:38:04.563 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-02-11T09:38:04.563 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:04.688 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2021-02-11T09:38:04.689 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2021-02-11T09:38:04.709 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-02-11T09:38:04.813 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2021-02-11T09:38:04.815 INFO:teuthology.task.ansible.out:son", "validate_certs": true} 2021-02-11T09:38:04.824 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:04.825 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:04.836 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-02-11T09:38:04.940 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-02-11T09:38:04.951 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-02-11T09:38:05.065 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2021-02-11T09:38:05.076 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2021-02-11T09:38:05.077 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-02-11T09:38:05.180 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-02-11T09:38:05.211 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-02-11T09:38:05.295 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-02-11T09:38:05.348 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "davidz", "validate_certs": true} 2021-02-11T09:38:05.421 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-02-11T09:38:05.494 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-02-11T09:38:05.546 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-02-11T09:38:05.619 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-02-11T09:38:05.661 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2021-02-11T09:38:05.663 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-02-11T09:38:05.745 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2021-02-11T09:38:05.777 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2021-02-11T09:38:05.778 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-02-11T09:38:05.880 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2021-02-11T09:38:05.882 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2021-02-11T09:38:05.902 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-02-11T09:38:06.016 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2021-02-11T09:38:06.018 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2021-02-11T09:38:06.027 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-02-11T09:38:06.142 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-02-11T09:38:06.153 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-02-11T09:38:06.267 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": nu 2021-02-11T09:38:06.268 INFO:teuthology.task.ansible.out:ll, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-02-11T09:38:06.278 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-02-11T09:38:06.404 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46 2021-02-11T09:38:06.405 INFO:teuthology.task.ansible.out:TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1u 2021-02-11T09:38:06.406 INFO:teuthology.task.ansible.out:gvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjEfdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE5 2021-02-11T09:38:06.408 INFO:teuthology.task.ansible.out:0xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+dTzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-02-11T09:38:06.519 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-02-11T09:38:06.519 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-02-11T09:38:06.644 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-02-11T09:38:06.645 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-02-11T09:38:06.759 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-02-11T09:38:06.760 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-02-11T09:38:06.885 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-02-11T09:38:06.916 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-02-11T09:38:07.000 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-02-11T09:38:07.033 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-02-11T09:38:07.127 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2021-02-11T09:38:07.128 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2021-02-11T09:38:07.169 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2021-02-11T09:38:07.170 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2021-02-11T09:38:07.242 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-02-11T09:38:07.295 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2021-02-11T09:38:07.367 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-02-11T09:38:07.440 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-02-11T09:38:07.493 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2021-02-11T09:38:07.494 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2021-02-11T09:38:07.556 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-02-11T09:38:07.608 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-02-11T09:38:07.671 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-02-11T09:38:07.735 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-02-11T09:38:07.819 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2021-02-11T09:38:07.861 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2021-02-11T09:38:07.863 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2021-02-11T09:38:07.966 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-02-11T09:38:07.978 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-02-11T09:38:08.096 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-02-11T09:38:08.117 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-02-11T09:38:08.211 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-02-11T09:38:08.254 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-02-11T09:38:08.255 INFO:teuthology.task.ansible.out: 2021-02-11T09:38:08.358 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-02-11T09:38:08.379 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-02-11T09:38:08.484 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2021-02-11T09:38:08.536 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2021-02-11T09:38:08.538 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-02-11T09:38:08.610 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2021-02-11T09:38:08.662 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2021-02-11T09:38:08.664 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-02-11T09:38:08.735 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2021-02-11T09:38:08.798 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-02-11T09:38:08.861 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-02-11T09:38:08.923 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-02-11T09:38:08.987 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2021-02-11T09:38:09.049 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-02-11T09:38:09.122 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2021-02-11T09:38:09.185 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": nu 2021-02-11T09:38:09.187 INFO:teuthology.task.ansible.out:ll, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-02-11T09:38:09.248 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-02-11T09:38:09.342 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2021-02-11T09:38:09.343 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2021-02-11T09:38:09.345 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-02-11T09:38:09.374 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-02-11T09:38:09.488 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-02-11T09:38:09.489 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-02-11T09:38:09.614 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2021-02-11T09:38:09.625 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-02-11T09:38:09.740 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2021-02-11T09:38:09.751 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-02-11T09:38:09.855 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-02-11T09:38:09.897 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-02-11T09:38:09.982 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-02-11T09:38:10.045 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-02-11T09:38:10.108 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-02-11T09:38:10.181 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2021-02-11T09:38:10.183 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2021-02-11T09:38:10.223 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-02-11T09:38:10.328 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-02-11T09:38:10.349 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2021-02-11T09:38:10.464 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-02-11T09:38:10.466 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1 2021-02-11T09:38:10.467 INFO:teuthology.task.ansible.out:lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2021-02-11T09:38:10.581 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2021-02-11T09:38:10.583 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2021-02-11T09:38:10.592 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-02-11T09:38:10.707 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-02-11T09:38:10.708 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-02-11T09:38:10.822 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-02-11T09:38:13.018 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/n 2021-02-11T09:38:13.020 INFO:teuthology.task.ansible.out:bQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/ 2021-02-11T09:38:13.021 INFO:teuthology.task.ansible.out:Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kkke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QR 2021-02-11T09:38:13.023 INFO:teuthology.task.ansible.out:FgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.localdomain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y 2021-02-11T09:38:13.024 INFO:teuthology.task.ansible.out:2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG0 2021-02-11T09:38:13.025 INFO:teuthology.task.ansible.out:2oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJk 2021-02-11T09:38:13.027 INFO:teuthology.task.ansible.out:u7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": 2021-02-11T09:38:13.028 INFO:teuthology.task.ansible.out: null, "state": "present", "user": "ideepika", "validate_certs": true} ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-02-11T09:38:13.029 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ720 2021-02-11T09:38:13.031 INFO:teuthology.task.ansible.out:9c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} ok: [smithi098.front.sepia.ce 2021-02-11T09:38:13.032 INFO:teuthology.task.ansible.out:ph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHy 2021-02-11T09:38:13.033 INFO:teuthology.task.ansible.out:UqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0d 2021-02-11T09:38:13.035 INFO:teuthology.task.ansible.out:XZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/autho 2021-02-11T09:38:13.036 INFO:teuthology.task.ansible.out:rized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat 2021-02-11T09:38:13.037 INFO:teuthology.task.ansible.out:", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKX 2021-02-11T09:38:13.039 INFO:teuthology.task.ansible.out:xnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNm 2021-02-11T09:38:13.040 INFO:teuthology.task.ansible.out:TD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.local", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO 2021-02-11T09:38:13.041 INFO:teuthology.task.ansible.out:3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": " 2021-02-11T09:38:13.043 INFO:teuthology.task.ansible.out:h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": " 2021-02-11T09:38:13.044 INFO:teuthology.task.ansible.out:shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true 2021-02-11T09:38:13.045 INFO:teuthology.task.ansible.out:, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "man 2021-02-11T09:38:13.047 INFO:teuthology.task.ansible.out:age_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul 2021-02-11T09:38:13.048 INFO:teuthology.task.ansible.out:6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8 2021-02-11T09:38:13.049 INFO:teuthology.task.ansible.out:313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-02-11T09:38:13.058 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-02-11T09:38:13.111 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2021-02-11T09:38:13.175 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-02-11T09:38:13.238 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2021-02-11T09:38:13.311 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-02-11T09:38:13.363 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2021-02-11T09:38:13.458 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-02-11T09:38:13.490 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2021-02-11T09:38:13.573 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2021-02-11T09:38:13.606 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2021-02-11T09:38:13.721 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2021-02-11T09:38:13.722 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1 2021-02-11T09:38:13.724 INFO:teuthology.task.ansible.out:lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2021-02-11T09:38:13.847 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2021-02-11T09:38:13.859 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-02-11T09:38:13.964 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2021-02-11T09:38:14.006 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-02-11T09:38:14.080 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2021-02-11T09:38:14.134 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-02-11T09:38:14.207 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2021-02-11T09:38:14.249 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2021-02-11T09:38:14.332 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2021-02-11T09:38:14.374 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.locald 2021-02-11T09:38:14.375 INFO:teuthology.task.ansible.out:omain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2021-02-11T09:38:14.448 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2021-02-11T09:38:14.521 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2021-02-11T09:38:14.522 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2021-02-11T09:38:14.573 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2021-02-11T09:38:14.656 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-02-11T09:38:14.688 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2021-02-11T09:38:14.792 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-02-11T09:38:14.803 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2021-02-11T09:38:14.918 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2021-02-11T09:38:14.929 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2021-02-11T09:38:15.035 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2021-02-11T09:38:15.056 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2021-02-11T09:38:15.160 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2021-02-11T09:38:15.203 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2021-02-11T09:38:15.279 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjL 2021-02-11T09:38:15.280 INFO:teuthology.task.ansible.out:d+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2021-02-11T09:38:15.332 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2021-02-11T09:38:15.384 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2021-02-11T09:38:15.468 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2021-02-11T09:38:15.511 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2021-02-11T09:38:15.594 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2021-02-11T09:38:15.596 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2021-02-11T09:38:15.626 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2021-02-11T09:38:15.731 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2021-02-11T09:38:15.752 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2021-02-11T09:38:15.856 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2021-02-11T09:38:15.867 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2021-02-11T09:38:15.992 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2021-02-11T09:38:15.993 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2021-02-11T09:38:16.108 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2021-02-11T09:38:16.129 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2021-02-11T09:38:16.223 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-02-11T09:38:16.265 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2021-02-11T09:38:16.338 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2021-02-11T09:38:16.401 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2021-02-11T09:38:16.454 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-02-11T09:38:16.537 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", 2021-02-11T09:38:16.539 INFO:teuthology.task.ansible.out:"manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-02-11T09:38:16.569 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2021-02-11T09:38:16.673 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2021-02-11T09:38:16.695 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQKiUAlB8dxudAY60M7jnIJvs9yb2pNaT+BVv1Vk1kMs0XjPjcw9cyT3DtQ1xGLF367phRi/cfQbXOYbHG+uyBzQ4EEnBFvziktWRx4mfeWYxKjkjbfkVfpKPsxztqCbwm6+NYzMxGK2cExN7jUHnghHAjfA0IUpVVg9peBD3kUveLwS+UxuJAnuj+G4480LS6GDyLrhFEXU7uOIIlW4S9oLnU6vJL6GAzwyLE4ijTA8Ek03SJ01Ey/gpU9AqnQNp9u12l/7sJely6sI9pAuIzmS8Ht324hexxSWN5sFjXgLvLfFK83ziDBdrFBr0WXnWsFdRpHs2CwSq1JTkByqbN hmunjulu@hmunjulu.remote.csb", "key_options": null, "keyfile": "/home/hmunjulu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hmunjulu", "validate_certs": true} 2021-02-11T09:38:16.809 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2021-02-11T09:38:16.810 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2021-02-11T09:38:16.936 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev", "key_options": null, "keyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2021-02-11T09:38:16.947 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2021-02-11T09:38:17.061 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2021-02-11T09:38:17.063 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2021-02-11T09:38:17.187 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2021-02-11T09:38:17.199 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2021-02-11T09:38:17.303 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com", "key_options": null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} 2021-02-11T09:38:17.324 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2021-02-11T09:38:17.429 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDr6H7hmtGCUrO3GppgMHwbmA4XT/Vipz/uLG/N8h6ZxtaLQKpCefRpdKcFlOdDwxkTmT5q9vmXQ8nl95lTGq+eeMxl4ho104ZD9e9AIYmgYqFy7/qNP8gkqduIPlypLL9m5rElTcGd7zqfMqgljMvAmnXLtZNJlIuFtszVU5WPI1QCwfQ3t5aHF6j66bpVoN9uWItVGB1LApQz9TtViWsQ9Hl8X4YauJToO+TkuSX+H/OGH7ZC6CPOzNZb3b3vF/jLMnTzXl86PJs1se4DjKkRGLVCVbr2WhLwQeZ8TEn3Hg1Gn5X8zNM5cTo6U8SnXBAe9ZYbuyfvghrNLdlZpmN+8l0/l5zC3z+N+sEzklXwHw4MQLF+KSloVFd9NN/kyenufTZAyp8VxllZ9TgdC+CPORIWyzmu1elOtYo7LNuE6CIh/Zm9B54SO6hVHhVoOCqe/dIyjeHJA0WKIZblK1XZEvWuu0E2T3jhxfe1GICyTukeB8WoBQH9DZ1yzMPt59E= nia@redhat.com", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2021-02-11T09:38:17.483 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2021-02-11T09:38:17.516 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:17 +0000 (0:00:14.207) 0:00:43.437 ***** 2021-02-11T09:38:17.555 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-02-11T09:38:17.557 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:17.567 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:17.577 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:17.578 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:17.589 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-11T09:38:17.590 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7 2021-02-11T09:38:17.591 INFO:teuthology.task.ansible.out:debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:17.601 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:17.612 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:17.612 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:17.717 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-11T09:38:17.717 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2021-02-11T09:38:17.739 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-02-11T09:38:17.740 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2021-02-11T09:38:17.750 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:17.760 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:17.855 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-11T09:38:17.856 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2021-02-11T09:38:17.877 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-02-11T09:38:17.878 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2021-02-11T09:38:17.991 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-11T09:38:17.993 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/Ib 2021-02-11T09:38:17.994 INFO:teuthology.task.ansible.out:jJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2021-02-11T09:38:18.003 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.004 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.015 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.016 INFO:teuthology.task.ansible.out: 2021-02-11T09:38:18.026 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.028 INFO:teuthology.task.ansible.out: 2021-02-11T09:38:18.037 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.039 INFO:teuthology.task.ansible.out: skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.049 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.050 INFO:teuthology.task.ansible.out: 2021-02-11T09:38:18.060 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.061 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.072 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.083 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.084 INFO:teuthology.task.ansible.out: skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.094 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.105 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.106 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.116 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.127 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.128 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.139 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-11T09:38:18.140 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.150 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.151 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.162 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.173 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.173 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.184 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.185 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.195 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.207 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.207 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.218 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.229 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.229 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.240 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.251 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.251 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.262 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.262 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.273 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.274 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-11T09:38:18.274 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2021-02-11T09:38:18.285 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-02-11T09:38:18.286 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.296 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.297 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.307 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.318 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.319 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.330 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.340 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.341 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.352 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.352 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.363 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.374 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.374 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.385 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.396 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.397 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.408 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-11T09:38:18.409 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.409 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.420 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.431 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.431 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.442 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.453 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.453 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.464 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.475 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.475 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.486 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.486 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.497 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.508 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.508 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.519 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.530 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-11T09:38:18.531 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57da 2021-02-11T09:38:18.532 INFO:teuthology.task.ansible.out:e1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.542 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.542 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.553 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.564 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.564 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.575 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.586 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.586 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.597 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.597 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.608 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.619 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.620 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.630 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.641 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.641 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.652 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.663 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-11T09:38:18.664 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b5 2021-02-11T09:38:18.666 INFO:teuthology.task.ansible.out:39a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.675 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.687 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.688 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.699 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.709 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.710 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.721 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.732 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.732 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.743 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.754 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.764 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.765 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.776 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.790 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.798 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-11T09:38:18.800 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2021-02-11T09:38:18.810 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-02-11T09:38:18.811 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.821 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.832 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.832 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.843 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.854 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.854 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.865 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.877 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.887 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.888 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.910 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.910 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.911 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:18.939 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-11T09:38:18.941 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2021-02-11T09:38:18.975 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:18 +0000 (0:00:01.459) 0:00:44.896 ***** 2021-02-11T09:38:19.004 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** 2021-02-11T09:38:19.006 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:19.041 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:19 +0000 (0:00:00.065) 0:00:44.962 ***** 2021-02-11T09:38:19.091 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-11T09:38:19.093 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjL 2021-02-11T09:38:19.094 INFO:teuthology.task.ansible.out:d+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2021-02-11T09:38:19.221 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2021-02-11T09:38:19.232 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T09:38:19.233 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2021-02-11T09:38:19.358 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-02-11T09:38:19.369 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-11T09:38:19.371 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2021-02-11T09:38:19.475 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T09:38:19.477 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-02-11T09:38:19.497 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-11T09:38:19.498 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2021-02-11T09:38:19.602 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T09:38:19.603 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-02-11T09:38:19.658 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-11T09:38:19.660 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2021-02-11T09:38:19.732 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T09:38:19.734 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-02-11T09:38:19.796 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-11T09:38:19.798 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2021-02-11T09:38:19.849 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T09:38:19.850 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-02-11T09:38:19.912 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-11T09:38:19.914 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2021-02-11T09:38:19.969 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T09:38:19.970 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-02-11T09:38:20.042 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-11T09:38:20.044 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2021-02-11T09:38:20.084 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T09:38:20.086 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-02-11T09:38:20.189 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-11T09:38:20.191 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-02-11T09:38:20.211 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T09:38:20.212 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2021-02-11T09:38:20.326 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-11T09:38:20.327 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2021-02-11T09:38:20.337 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T09:38:20.339 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-02-11T09:38:20.442 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-11T09:38:20.443 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-02-11T09:38:20.453 INFO:teuthology.task.ansible.out: 2021-02-11T09:38:20.454 INFO:teuthology.task.ansible.out:TASK [users : Remove revoked users] ******************************************** ok: [smithi094.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-02-11T09:38:20.557 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-11T09:38:20.559 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2021-02-11T09:38:20.580 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T09:38:20.580 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2021-02-11T09:38:20.685 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-11T09:38:20.687 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQKiUAlB8dxudAY60M7jnIJvs9yb2pNaT+BVv1Vk1kMs0XjPjcw9cyT3DtQ1xGLF367phRi/cfQbXOYbHG+uyBzQ4EEnBFvziktWRx4mfeWYxKjkjbfkVfpKPsxztqCbwm6+NYzMxGK2cExN7jUHnghHAjfA0IUpVVg9peBD3kUveLwS+UxuJAnuj+G4480LS6GDyLrhFEXU7uOIIlW4S9oLnU6vJL6GAzwyLE4ijTA8Ek03SJ01Ey/gpU9AqnQNp9u12l/7sJely6sI9pAuIzmS8Ht324hexxSWN5sFjXgLvLfFK83ziDBdrFBr0WXnWsFdRpHs2CwSq1JTkByqbN hmunjulu@hmunjulu.remote.csb", "key_options": null, "keyfile": "/home/hmunjulu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hmunjulu", "validate_certs": true} 2021-02-11T09:38:20.697 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T09:38:20.698 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-02-11T09:38:20.812 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-11T09:38:20.813 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2021-02-11T09:38:20.823 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T09:38:20.824 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2021-02-11T09:38:20.938 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-02-11T09:38:20.949 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-11T09:38:20.951 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev", "key_options": null, "keyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2021-02-11T09:38:21.066 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T09:38:21.068 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2021-02-11T09:38:21.192 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T09:38:21.193 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2021-02-11T09:38:21.307 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T09:38:21.309 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-02-11T09:38:21.329 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-11T09:38:21.331 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com", "key_options": null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} 2021-02-11T09:38:21.434 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T09:38:21.435 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-02-11T09:38:21.458 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-11T09:38:21.459 INFO:teuthology.task.ansible.out:changed: [smithi098.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDr6H7hmtGCUrO3GppgMHwbmA4XT/Vipz/uLG/N8h6ZxtaLQKpCefRpdKcFlOdDwxkTmT5q9vmXQ8nl95lTGq+eeMxl4ho104ZD9e9AIYmgYqFy7/qNP8gkqduIPlypLL9m5rElTcGd7zqfMqgljMvAmnXLtZNJlIuFtszVU5WPI1QCwfQ3t5aHF6j66bpVoN9uWItVGB1LApQz9TtViWsQ9Hl8X4YauJToO+TkuSX+H/OGH7ZC6CPOzNZb3b3vF/jLMnTzXl86PJs1se4DjKkRGLVCVbr2WhLwQeZ8TEn3Hg1Gn5X8zNM5cTo6U8SnXBAe9ZYbuyfvghrNLdlZpmN+8l0/l5zC3z+N+sEzklXwHw4MQLF+KSloVFd9NN/kyenufTZAyp8VxllZ9TgdC+CPORIWyzmu1elOtYo7LNuE6CIh/Zm9B54SO6hVHhVoOCqe/dIyjeHJA0WKIZblK1XZEvWuu0E2T3jhxfe1GICyTukeB8WoBQH9DZ1yzMPt59E= nia@redhat.com", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2021-02-11T09:38:21.535 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:21 +0000 (0:00:02.493) 0:00:47.455 ***** 2021-02-11T09:38:21.551 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T09:38:21.552 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-02-11T09:38:21.573 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:21.584 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:21.595 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:21.606 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:21.606 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:21.617 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:21.628 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:21.638 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:21.649 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:21.671 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T09:38:21.672 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-02-11T09:38:21.785 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-02-11T09:38:21.787 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} TASK [users : Remove revoked users] ******************************************** ok: [smithi094.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item 2021-02-11T09:38:21.788 INFO:teuthology.task.ansible.out:", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2021-02-11T09:38:21.797 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-02-11T09:38:21.799 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:21.808 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:21.913 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T09:38:21.915 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-02-11T09:38:21.945 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-02-11T09:38:21.947 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2021-02-11T09:38:22.029 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T09:38:22.030 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-02-11T09:38:22.061 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-02-11T09:38:22.062 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2021-02-11T09:38:22.072 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.072 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.083 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.094 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.105 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.116 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.116 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.127 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.138 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.149 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T09:38:22.151 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.161 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.162 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.172 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.183 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.183 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.194 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.205 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.216 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.217 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.228 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.239 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.249 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.260 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.261 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T09:38:22.261 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-02-11T09:38:22.262 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.273 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.283 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.284 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.295 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.306 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.316 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.317 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.328 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.338 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.339 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.350 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.360 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.361 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.372 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.383 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.394 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.395 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T09:38:22.396 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-02-11T09:38:22.396 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.407 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.418 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.419 INFO:teuthology.task.ansible.out: skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.429 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.441 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.451 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.452 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.463 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.473 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.474 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.495 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.506 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.507 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.508 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T09:38:22.510 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-02-11T09:38:22.520 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-02-11T09:38:22.521 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.531 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.542 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.552 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.564 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.564 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.575 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.586 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.586 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.597 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.597 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.609 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.619 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.620 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.631 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T09:38:22.633 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.643 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.654 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.665 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.676 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.686 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.687 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.698 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.708 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.709 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.720 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.730 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.731 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.741 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.752 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.753 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.764 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T09:38:22.765 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.776 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.787 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.798 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.799 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.809 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.823 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.824 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.835 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.846 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.857 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.857 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.868 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.868 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.879 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.880 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T09:38:22.882 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-02-11T09:38:22.892 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-02-11T09:38:22.893 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.904 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.915 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.915 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.926 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.937 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.937 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.948 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.959 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.960 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.971 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.981 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.982 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:22.993 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:23.004 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:23.004 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T09:38:23.004 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-02-11T09:38:23.016 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-02-11T09:38:23.017 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:23.028 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:23.028 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:23.039 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:23.050 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:23.051 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:23.062 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:23.119 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:23 +0000 (0:00:01.584) 0:00:49.039 ***** 2021-02-11T09:38:23.136 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T09:38:23.136 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-02-11T09:38:23.147 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:23.183 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:23 +0000 (0:00:00.063) 0:00:49.103 ***** 2021-02-11T09:38:23.251 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T09:38:23.252 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-02-11T09:38:23.336 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2021-02-11T09:38:23.368 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-02-11T09:38:23.463 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-02-11T09:38:23.485 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-02-11T09:38:23.590 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-02-11T09:38:23.601 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-02-11T09:38:23.706 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-02-11T09:38:23.718 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "name": "dorinda", "state": "absent"} 2021-02-11T09:38:23.834 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-02-11T09:38:23.959 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-02-11T09:38:24.073 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-02-11T09:38:24.219 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-02-11T09:38:24.240 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "zyan", "name": "zyan", "remove": false, "state": "absent"} 2021-02-11T09:38:24.334 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2021-02-11T09:38:24.355 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-02-11T09:38:24.450 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-02-11T09:38:24.482 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-02-11T09:38:24.587 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-02-11T09:38:24.598 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-02-11T09:38:24.714 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2021-02-11T09:38:24.715 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-02-11T09:38:24.829 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-02-11T09:38:24.841 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-02-11T09:38:24.955 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-02-11T09:38:24.977 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2021-02-11T09:38:25.082 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-02-11T09:38:25.093 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-02-11T09:38:25.197 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-02-11T09:38:25.239 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-02-11T09:38:25.313 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-02-11T09:38:25.355 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2021-02-11T09:38:25.438 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-02-11T09:38:25.494 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:25 +0000 (0:00:02.311) 0:00:51.414 ***** 2021-02-11T09:38:25.510 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-02-11T09:38:25.615 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-02-11T09:38:25.739 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-02-11T09:38:25.844 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2021-02-11T09:38:25.846 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => {"changed": true, "checksum": "5ebc9fcf8859a9c716ab10089a7bec7e44d29368", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "ee06ed012f51c2e3764171d4b9180349", "mode": "0644", "owner": "root", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1613036305.5156977-235686502648562/source", "state": "file", "uid": 0} 2021-02-11T09:38:25.880 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:25 +0000 (0:00:00.385) 0:00:51.800 ***** 2021-02-11T09:38:25.886 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T09:38:25.886 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-02-11T09:38:25.908 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2021-02-11T09:38:25.909 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => { "msg": "Host smithi094.front.sepia.ceph.com is running Ubuntu 18.04 (bionic)" } 2021-02-11T09:38:25.945 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:25 +0000 (0:00:00.065) 0:00:51.866 ***** 2021-02-11T09:38:25.972 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T09:38:25.973 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2021-02-11T09:38:25.983 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2021-02-11T09:38:25.984 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2021-02-11T09:38:26.018 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:26 +0000 (0:00:00.073) 0:00:51.939 ***** 2021-02-11T09:38:26.046 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2021-02-11T09:38:26.047 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:26.080 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:26 +0000 (0:00:00.061) 0:00:52.000 ***** 2021-02-11T09:38:26.086 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T09:38:26.086 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-02-11T09:38:26.108 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:26.144 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:26 +0000 (0:00:00.064) 0:00:52.065 ***** 2021-02-11T09:38:26.171 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2021-02-11T09:38:26.173 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:26.209 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:26 +0000 (0:00:00.065) 0:00:52.130 ***** 2021-02-11T09:38:26.216 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T09:38:26.217 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-02-11T09:38:26.239 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:26.273 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:26 +0000 (0:00:00.063) 0:00:52.194 ***** 2021-02-11T09:38:26.300 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:26.335 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:26 +0000 (0:00:00.062) 0:00:52.256 ***** 2021-02-11T09:38:26.362 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:26.397 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:26 +0000 (0:00:00.061) 0:00:52.317 ***** 2021-02-11T09:38:26.403 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T09:38:26.404 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-02-11T09:38:26.415 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:26.450 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:26 +0000 (0:00:00.052) 0:00:52.370 ***** 2021-02-11T09:38:26.476 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:26.510 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:26 +0000 (0:00:00.060) 0:00:52.430 ***** 2021-02-11T09:38:26.526 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T09:38:26.528 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-02-11T09:38:26.548 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** skipping: [smithi094.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-11T09:38:26.549 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-11T09:38:26.583 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:26 +0000 (0:00:00.073) 0:00:52.504 ***** 2021-02-11T09:38:26.610 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2021-02-11T09:38:26.611 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:26.647 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:26 +0000 (0:00:00.063) 0:00:52.568 ***** 2021-02-11T09:38:26.653 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T09:38:26.654 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-02-11T09:38:26.676 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2021-02-11T09:38:26.677 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:26.709 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:26 +0000 (0:00:00.062) 0:00:52.630 ***** 2021-02-11T09:38:26.735 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2021-02-11T09:38:26.737 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-11T09:38:26.770 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:26 +0000 (0:00:00.060) 0:00:52.690 ***** 2021-02-11T09:38:26.786 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T09:38:26.786 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-02-11T09:38:26.798 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:26.833 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:26 +0000 (0:00:00.063) 0:00:52.754 ***** 2021-02-11T09:38:26.860 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** skipping: [smithi094.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-11T09:38:26.896 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:26 +0000 (0:00:00.063) 0:00:52.817 ***** 2021-02-11T09:38:26.923 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T09:38:26.924 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} TASK [common : Set rhsm_registered if we just registered] ********************** skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:26.958 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:26 +0000 (0:00:00.061) 0:00:52.879 ***** 2021-02-11T09:38:26.985 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2021-02-11T09:38:26.986 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:27.018 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:27 +0000 (0:00:00.060) 0:00:52.939 ***** 2021-02-11T09:38:27.045 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2021-02-11T09:38:27.046 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:27.082 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:27 +0000 (0:00:00.060) 0:00:53.000 ***** 2021-02-11T09:38:27.085 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T09:38:27.085 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-02-11T09:38:27.107 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2021-02-11T09:38:27.108 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:27.140 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:27 +0000 (0:00:00.060) 0:00:53.060 ***** 2021-02-11T09:38:27.165 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2021-02-11T09:38:27.166 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-11T09:38:27.199 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:27 +0000 (0:00:00.059) 0:00:53.120 ***** 2021-02-11T09:38:27.205 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T09:38:27.205 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-02-11T09:38:27.217 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-02-11T09:38:27.218 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:27.251 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:27 +0000 (0:00:00.051) 0:00:53.171 ***** 2021-02-11T09:38:27.280 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2021-02-11T09:38:27.281 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:27.315 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:27 +0000 (0:00:00.063) 0:00:53.235 ***** 2021-02-11T09:38:27.321 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T09:38:27.321 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-02-11T09:38:27.333 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2021-02-11T09:38:27.334 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:27.368 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:27 +0000 (0:00:00.053) 0:00:53.288 ***** 2021-02-11T09:38:27.395 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2021-02-11T09:38:27.396 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:27.429 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:27 +0000 (0:00:00.061) 0:00:53.350 ***** 2021-02-11T09:38:27.446 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T09:38:27.447 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-02-11T09:38:27.457 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2021-02-11T09:38:27.458 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:27.491 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:27 +0000 (0:00:00.062) 0:00:53.412 ***** 2021-02-11T09:38:27.518 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2021-02-11T09:38:27.519 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:27.550 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:27 +0000 (0:00:00.059) 0:00:53.471 ***** 2021-02-11T09:38:27.566 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T09:38:27.567 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-02-11T09:38:27.577 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2021-02-11T09:38:27.579 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-11T09:38:27.612 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:27 +0000 (0:00:00.061) 0:00:53.532 ***** 2021-02-11T09:38:27.642 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:27.676 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:27 +0000 (0:00:00.063) 0:00:53.596 ***** 2021-02-11T09:38:27.692 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T09:38:27.694 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} TASK [common : Remove old apt-mirror repository definition.] ******************* skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:27.728 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:27 +0000 (0:00:00.052) 0:00:53.649 ***** 2021-02-11T09:38:27.754 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2021-02-11T09:38:27.756 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:27.788 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:27 +0000 (0:00:00.059) 0:00:53.709 ***** 2021-02-11T09:38:27.815 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T09:38:27.817 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-02-11T09:38:27.827 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** skipping: [smithi094.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:27.827 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:27.862 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:27 +0000 (0:00:00.074) 0:00:53.783 ***** 2021-02-11T09:38:27.889 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-02-11T09:38:27.890 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-mirrorlist", "skip_reason": "Conditional result was False"} 2021-02-11T09:38:27.900 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-testing-mirrorlist", "skip_reason": "Conditional result was False"} 2021-02-11T09:38:27.936 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:27 +0000 (0:00:00.073) 0:00:53.856 ***** 2021-02-11T09:38:27.942 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T09:38:27.942 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-02-11T09:38:27.964 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:27.997 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:27 +0000 (0:00:00.061) 0:00:53.918 ***** 2021-02-11T09:38:28.058 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T09:38:28.059 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "name": "dorinda", "state": "absent"} 2021-02-11T09:38:28.152 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-02-11T09:38:28.153 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "cmd": "cat /etc/timezone", "delta": "0:00:00.022495", "end": "2021-02-11 09:38:28.128550", "rc": 0, "start": "2021-02-11 09:38:28.106055", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-02-11T09:38:28.185 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:28 +0000 (0:00:00.188) 0:00:54.106 ***** 2021-02-11T09:38:28.212 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2021-02-11T09:38:28.213 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:28.247 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:28 +0000 (0:00:00.061) 0:00:54.167 ***** 2021-02-11T09:38:28.274 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2021-02-11T09:38:28.275 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:28.308 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:28 +0000 (0:00:00.060) 0:00:54.228 ***** 2021-02-11T09:38:28.334 INFO:teuthology.task.ansible.out: TASK [common : Mask sleep units] *********************************************** 2021-02-11T09:38:28.335 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "skip_reason": "Conditional result was False"} 2021-02-11T09:38:28.345 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "skip_reason": "Conditional result was False"} 2021-02-11T09:38:28.356 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "skip_reason": "Conditional result was False"} 2021-02-11T09:38:28.367 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "skip_reason": "Conditional result was False"} 2021-02-11T09:38:28.400 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:28 +0000 (0:00:00.092) 0:00:54.321 ***** 2021-02-11T09:38:28.427 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T09:38:28.429 INFO:teuthology.task.ansible.out:changed: [smithi098.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "zyan", "name": "zyan", "remove": false, "state": "absent"} TASK [common : Get the current timezone] *************************************** skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:28.464 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:28 +0000 (0:00:00.064) 0:00:54.385 ***** 2021-02-11T09:38:28.501 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2021-02-11T09:38:28.502 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:28.538 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:28 +0000 (0:00:00.074) 0:00:54.459 ***** 2021-02-11T09:38:28.545 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T09:38:28.545 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-02-11T09:38:28.567 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2021-02-11T09:38:28.568 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:28.602 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:28 +0000 (0:00:00.063) 0:00:54.523 ***** 2021-02-11T09:38:28.628 INFO:teuthology.task.ansible.out: 2021-02-11T09:38:28.628 INFO:teuthology.task.ansible.out:TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2021-02-11T09:38:28.630 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:28.664 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:28 +0000 (0:00:00.062) 0:00:54.585 ***** 2021-02-11T09:38:28.680 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T09:38:28.682 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-02-11T09:38:28.692 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:28.727 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:28 +0000 (0:00:00.063) 0:00:54.648 ***** 2021-02-11T09:38:28.795 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T09:38:28.797 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-02-11T09:38:28.933 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-02-11T09:38:29.048 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-02-11T09:38:29.186 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-02-11T09:38:29.310 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-02-11T09:38:29.436 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-02-11T09:38:29.519 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2021-02-11T09:38:29.520 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.669662", "end": "2021-02-11 09:38:29.500380", "rc": 0, "start": "2021-02-11 09:38:28.830718", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-11T09:38:29.557 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:29 +0000 (0:00:00.829) 0:00:55.477 ***** 2021-02-11T09:38:29.563 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T09:38:29.563 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-02-11T09:38:29.575 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2021-02-11T09:38:29.576 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:29.609 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:29 +0000 (0:00:00.052) 0:00:55.530 ***** 2021-02-11T09:38:29.676 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T09:38:29.678 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-02-11T09:38:29.730 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:29 +0000 (0:00:00.121) 0:00:55.651 ***** 2021-02-11T09:38:29.777 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2021-02-11T09:38:29.779 INFO:teuthology.task.ansible.out:[WARNING]: Consider using the apt module rather than running 'apt-get'. If you need to use command because apt is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi094.front.sepia.ceph.com] => {"changed": true, "cmd": ["apt-get", "clean"], "delta": "0:00:00.037236", "end": "2021-02-11 09:38:29.751061", "rc": 0, "start": "2021-02-11 09:38:29.713825", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-11T09:38:29.813 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:29 +0000 (0:00:00.083) 0:00:55.734 ***** 2021-02-11T09:38:30.027 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2021-02-11T09:38:30.029 INFO:teuthology.task.ansible.out:changed: [smithi098.front.sepia.ceph.com] => {"changed": true, "checksum": "5ebc9fcf8859a9c716ab10089a7bec7e44d29368", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "ee06ed012f51c2e3764171d4b9180349", "mode": "0644", "owner": "root", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1613036309.750963-77913821238352/source", "state": "file", "uid": 0} 2021-02-11T09:38:30.063 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:30 +0000 (0:00:00.249) 0:00:55.984 ***** 2021-02-11T09:38:30.089 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2021-02-11T09:38:30.090 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => { "msg": "Host smithi098.front.sepia.ceph.com is running Ubuntu 18.04 (bionic)" } 2021-02-11T09:38:30.125 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:30 +0000 (0:00:00.062) 0:00:56.046 ***** 2021-02-11T09:38:30.162 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2021-02-11T09:38:30.163 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2021-02-11T09:38:30.201 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:30 +0000 (0:00:00.075) 0:00:56.121 ***** 2021-02-11T09:38:30.227 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2021-02-11T09:38:30.229 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:30.264 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:30 +0000 (0:00:00.063) 0:00:56.184 ***** 2021-02-11T09:38:30.290 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2021-02-11T09:38:30.292 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:30.325 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:30 +0000 (0:00:00.061) 0:00:56.246 ***** 2021-02-11T09:38:30.352 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2021-02-11T09:38:30.354 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:30.388 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:30 +0000 (0:00:00.062) 0:00:56.308 ***** 2021-02-11T09:38:30.414 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2021-02-11T09:38:30.416 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:30.450 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:30 +0000 (0:00:00.061) 0:00:56.370 ***** 2021-02-11T09:38:30.479 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2021-02-11T09:38:30.480 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:30.515 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:30 +0000 (0:00:00.065) 0:00:56.436 ***** 2021-02-11T09:38:30.546 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2021-02-11T09:38:30.547 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:30.587 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:30 +0000 (0:00:00.071) 0:00:56.508 ***** 2021-02-11T09:38:30.614 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2021-02-11T09:38:30.615 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:30.651 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:30 +0000 (0:00:00.063) 0:00:56.572 ***** 2021-02-11T09:38:30.678 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2021-02-11T09:38:30.680 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:30.715 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:30 +0000 (0:00:00.063) 0:00:56.635 ***** 2021-02-11T09:38:30.752 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2021-02-11T09:38:30.753 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi098.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-11T09:38:30.789 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:30 +0000 (0:00:00.073) 0:00:56.709 ***** 2021-02-11T09:38:30.819 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2021-02-11T09:38:30.821 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:30.855 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:30 +0000 (0:00:00.066) 0:00:56.776 ***** 2021-02-11T09:38:30.882 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2021-02-11T09:38:30.883 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:30.935 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:30 +0000 (0:00:00.079) 0:00:56.855 ***** 2021-02-11T09:38:30.962 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2021-02-11T09:38:30.963 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-11T09:38:31.015 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:31 +0000 (0:00:00.080) 0:00:56.936 ***** 2021-02-11T09:38:31.042 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2021-02-11T09:38:31.044 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:31.079 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:31 +0000 (0:00:00.063) 0:00:56.999 ***** 2021-02-11T09:38:31.105 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2021-02-11T09:38:31.107 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-11T09:38:31.140 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:31 +0000 (0:00:00.061) 0:00:57.061 ***** 2021-02-11T09:38:31.167 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:31.205 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:31 +0000 (0:00:00.064) 0:00:57.126 ***** 2021-02-11T09:38:31.233 INFO:teuthology.task.ansible.out: 2021-02-11T09:38:31.234 INFO:teuthology.task.ansible.out:TASK [common : List CDN releases available to system] ************************** skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:31.269 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:31 +0000 (0:00:00.063) 0:00:57.189 ***** 2021-02-11T09:38:31.295 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2021-02-11T09:38:31.297 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:31.333 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:31 +0000 (0:00:00.064) 0:00:57.253 ***** 2021-02-11T09:38:31.360 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2021-02-11T09:38:31.361 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:31.397 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:31 +0000 (0:00:00.063) 0:00:57.317 ***** 2021-02-11T09:38:31.426 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2021-02-11T09:38:31.428 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-11T09:38:31.463 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:31 +0000 (0:00:00.066) 0:00:57.384 ***** 2021-02-11T09:38:31.490 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:31.526 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:31 +0000 (0:00:00.062) 0:00:57.447 ***** 2021-02-11T09:38:31.553 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2021-02-11T09:38:31.555 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:31.592 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:31 +0000 (0:00:00.065) 0:00:57.512 ***** 2021-02-11T09:38:31.619 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2021-02-11T09:38:31.620 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:31.665 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:31 +0000 (0:00:00.073) 0:00:57.585 ***** 2021-02-11T09:38:31.691 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2021-02-11T09:38:31.693 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:31.730 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:31 +0000 (0:00:00.065) 0:00:57.651 ***** 2021-02-11T09:38:31.756 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2021-02-11T09:38:31.758 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:31.793 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:31 +0000 (0:00:00.063) 0:00:57.714 ***** 2021-02-11T09:38:31.820 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2021-02-11T09:38:31.821 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:31.857 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:31 +0000 (0:00:00.063) 0:00:57.777 ***** 2021-02-11T09:38:31.883 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2021-02-11T09:38:31.884 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-11T09:38:31.919 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:31 +0000 (0:00:00.061) 0:00:57.839 ***** 2021-02-11T09:38:31.945 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2021-02-11T09:38:31.947 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:31.980 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:31 +0000 (0:00:00.061) 0:00:57.901 ***** 2021-02-11T09:38:32.006 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2021-02-11T09:38:32.007 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:32.041 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:32 +0000 (0:00:00.061) 0:00:57.962 ***** 2021-02-11T09:38:32.068 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2021-02-11T09:38:32.069 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:32.103 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:32 +0000 (0:00:00.061) 0:00:58.023 ***** 2021-02-11T09:38:32.139 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2021-02-11T09:38:32.141 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} skipping: [smithi098.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:32.176 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:32 +0000 (0:00:00.073) 0:00:58.097 ***** 2021-02-11T09:38:32.213 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-02-11T09:38:32.215 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-mirrorlist", "skip_reason": "Conditional result was False"} skipping: [smithi098.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-testing-mirrorlist", "skip_reason": "Conditional result was False"} 2021-02-11T09:38:32.249 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:32 +0000 (0:00:00.072) 0:00:58.170 ***** 2021-02-11T09:38:32.274 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-02-11T09:38:32.276 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:32.314 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:32 +0000 (0:00:00.064) 0:00:58.234 ***** 2021-02-11T09:38:32.466 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-02-11T09:38:32.467 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => {"changed": false, "cmd": "cat /etc/timezone", "delta": "0:00:00.014906", "end": "2021-02-11 09:38:32.731069", "rc": 0, "start": "2021-02-11 09:38:32.716163", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-02-11T09:38:32.506 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:32 +0000 (0:00:00.191) 0:00:58.426 ***** 2021-02-11T09:38:32.532 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2021-02-11T09:38:32.533 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:32.568 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:32 +0000 (0:00:00.061) 0:00:58.488 ***** 2021-02-11T09:38:32.605 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2021-02-11T09:38:32.606 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:32.653 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:32 +0000 (0:00:00.085) 0:00:58.574 ***** 2021-02-11T09:38:32.690 INFO:teuthology.task.ansible.out: TASK [common : Mask sleep units] *********************************************** 2021-02-11T09:38:32.692 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "skip_reason": "Conditional result was False"} skipping: [smithi098.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "skip_reason": "Conditional result was False"} 2021-02-11T09:38:32.702 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "skip_reason": "Conditional result was False"} 2021-02-11T09:38:32.713 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "skip_reason": "Conditional result was False"} 2021-02-11T09:38:32.750 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:32 +0000 (0:00:00.096) 0:00:58.671 ***** 2021-02-11T09:38:32.776 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-02-11T09:38:32.778 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:32.818 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:32 +0000 (0:00:00.067) 0:00:58.739 ***** 2021-02-11T09:38:32.847 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2021-02-11T09:38:32.848 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:32.883 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:32 +0000 (0:00:00.064) 0:00:58.803 ***** 2021-02-11T09:38:32.909 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2021-02-11T09:38:32.910 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:32.950 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:32 +0000 (0:00:00.067) 0:00:58.871 ***** 2021-02-11T09:38:32.977 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2021-02-11T09:38:32.978 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:33.011 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:33 +0000 (0:00:00.060) 0:00:58.932 ***** 2021-02-11T09:38:33.037 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2021-02-11T09:38:33.038 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:33.072 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:33 +0000 (0:00:00.060) 0:00:58.993 ***** 2021-02-11T09:38:33.234 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2021-02-11T09:38:33.236 INFO:teuthology.task.ansible.out:changed: [smithi098.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.049016", "end": "2021-02-11 09:38:33.500404", "rc": 0, "start": "2021-02-11 09:38:33.451388", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-11T09:38:33.270 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:33 +0000 (0:00:00.197) 0:00:59.190 ***** 2021-02-11T09:38:33.296 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2021-02-11T09:38:33.298 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:33.333 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:33 +0000 (0:00:00.062) 0:00:59.253 ***** 2021-02-11T09:38:33.538 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2021-02-11T09:38:33.538 INFO:teuthology.task.ansible.out:changed: [smithi098.front.sepia.ceph.com] => {"changed": true, "cmd": ["apt-get", "clean"], "delta": "0:00:00.058389", "end": "2021-02-11 09:38:33.794084", "rc": 0, "start": "2021-02-11 09:38:33.735695", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-11T09:38:33.576 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:33 +0000 (0:00:00.243) 0:00:59.497 ***** 2021-02-11T09:38:33.749 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2021-02-11T09:38:33.751 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"attempts": 1, "cache_update_time": 1613036022, "cache_updated": false, "changed": false} 2021-02-11T09:38:33.793 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:33 +0000 (0:00:00.216) 0:00:59.713 ***** 2021-02-11T09:38:34.671 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2021-02-11T09:38:34.673 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"cache_update_time": 1613036022, "cache_updated": false, "changed": false} 2021-02-11T09:38:34.707 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:34 +0000 (0:00:00.914) 0:01:00.628 ***** 2021-02-11T09:38:34.734 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:34.769 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:34 +0000 (0:00:00.060) 0:01:00.689 ***** 2021-02-11T09:38:35.370 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2021-02-11T09:38:35.372 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "size": 131, "state": "file", "uid": 0} 2021-02-11T09:38:35.406 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:35 +0000 (0:00:00.637) 0:01:01.327 ***** 2021-02-11T09:38:36.252 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2021-02-11T09:38:36.254 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"cache_update_time": 1613036022, "cache_updated": false, "changed": false} 2021-02-11T09:38:36.288 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:36 +0000 (0:00:00.881) 0:01:02.209 ***** 2021-02-11T09:38:36.615 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-02-11T09:38:36.617 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "size": 2716736, "state": "file", "uid": 0} 2021-02-11T09:38:36.896 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "size": 1952038, "state": "file", "uid": 0} 2021-02-11T09:38:37.161 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "size": 580432, "state": "file", "uid": 0} 2021-02-11T09:38:37.162 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2021-02-11T09:38:37.162 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => {"attempts": 1, "cache_update_time": 1613036065, "cache_updated": false, "changed": false} 2021-02-11T09:38:37.164 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-02-11T09:38:37.196 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:37 +0000 (0:00:00.908) 0:01:03.117 ***** 2021-02-11T09:38:37.230 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:37 +0000 (0:00:00.033) 0:01:03.150 ***** 2021-02-11T09:38:37.310 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2021-02-11T09:38:37.311 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "size": 4096, "state": "directory", "uid": 0} 2021-02-11T09:38:37.344 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:37 +0000 (0:00:00.114) 0:01:03.264 ***** 2021-02-11T09:38:37.631 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-02-11T09:38:37.633 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "size": 13355, "state": "file", "uid": 0} 2021-02-11T09:38:37.892 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "size": 8602, "state": "file", "uid": 0} 2021-02-11T09:38:38.079 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2021-02-11T09:38:38.081 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => {"cache_update_time": 1613036065, "cache_updated": false, "changed": false} 2021-02-11T09:38:38.114 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:38 +0000 (0:00:00.769) 0:01:04.034 ***** 2021-02-11T09:38:38.140 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:38.175 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:38 +0000 (0:00:00.061) 0:01:04.096 ***** 2021-02-11T09:38:38.180 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-02-11T09:38:38.181 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "size": 3605, "state": "file", "uid": 0} 2021-02-11T09:38:38.221 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:38 +0000 (0:00:00.045) 0:01:04.142 ***** 2021-02-11T09:38:38.258 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2021-02-11T09:38:38.259 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib/nagios/plugins", "nrpe_group": "nagios", "nrpe_packages": ["nagios-nrpe-server", "nagios-plugins-basic"], "nrpe_service_name": "nagios-nrpe-server", "nrpe_user": "nagios"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/apt_systems.yml"], "changed": false} 2021-02-11T09:38:38.292 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:38 +0000 (0:00:00.071) 0:01:04.213 ***** 2021-02-11T09:38:38.318 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2021-02-11T09:38:38.320 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:38.352 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:38 +0000 (0:00:00.060) 0:01:04.273 ***** 2021-02-11T09:38:38.378 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2021-02-11T09:38:38.380 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-11T09:38:38.412 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:38 +0000 (0:00:00.059) 0:01:04.333 ***** 2021-02-11T09:38:38.449 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2021-02-11T09:38:38.450 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-02-11T09:38:38.460 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-02-11T09:38:38.470 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-02-11T09:38:38.471 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-02-11T09:38:38.506 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:38 +0000 (0:00:00.093) 0:01:04.426 ***** 2021-02-11T09:38:38.512 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2021-02-11T09:38:38.512 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "size": 131, "state": "file", "uid": 0} 2021-02-11T09:38:38.552 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:38 +0000 (0:00:00.045) 0:01:04.472 ***** 2021-02-11T09:38:38.557 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2021-02-11T09:38:38.559 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:38.593 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:38 +0000 (0:00:00.041) 0:01:04.514 ***** 2021-02-11T09:38:39.369 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2021-02-11T09:38:39.371 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => {"cache_update_time": 1613036065, "cache_updated": false, "changed": false} 2021-02-11T09:38:39.407 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:39 +0000 (0:00:00.814) 0:01:05.328 ***** 2021-02-11T09:38:39.455 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2021-02-11T09:38:39.457 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"cache_update_time": 1613036022, "cache_updated": false, "changed": false} 2021-02-11T09:38:39.492 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:39 +0000 (0:00:00.085) 0:01:05.413 ***** 2021-02-11T09:38:39.737 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-02-11T09:38:39.739 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "size": 2716736, "state": "file", "uid": 0} 2021-02-11T09:38:39.790 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2021-02-11T09:38:39.792 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "checksum": "4880e187ff9f437cc33858eb1d0fa49a06d67be4", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "size": 180, "state": "file", "uid": 0} 2021-02-11T09:38:39.825 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:39 +0000 (0:00:00.332) 0:01:05.746 ***** 2021-02-11T09:38:40.020 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-02-11T09:38:40.021 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "size": 1952038, "state": "file", "uid": 0} 2021-02-11T09:38:40.115 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2021-02-11T09:38:40.116 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/nagios/plugins/check_mem.sh", "size": 1964, "state": "file", "uid": 0} 2021-02-11T09:38:40.156 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:40 +0000 (0:00:00.330) 0:01:06.077 ***** 2021-02-11T09:38:40.291 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-02-11T09:38:40.292 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "size": 580432, "state": "file", "uid": 0} 2021-02-11T09:38:40.294 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-02-11T09:38:40.295 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "DAEMON_OPTS", "msg": ""} 2021-02-11T09:38:40.331 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:40 +0000 (0:00:00.174) 0:01:06.251 ***** 2021-02-11T09:38:40.402 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "NRPE_OPTS", "msg": ""} 2021-02-11T09:38:40.440 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:40 +0000 (0:00:00.109) 0:01:06.361 ***** 2021-02-11T09:38:40.457 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2021-02-11T09:38:40.458 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "size": 4096, "state": "directory", "uid": 0} 2021-02-11T09:38:40.500 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:40 +0000 (0:00:00.059) 0:01:06.421 ***** 2021-02-11T09:38:40.506 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2021-02-11T09:38:40.508 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:40.543 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:40 +0000 (0:00:00.042) 0:01:06.464 ***** 2021-02-11T09:38:40.570 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** skipping: [smithi094.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-11T09:38:40.604 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:40 +0000 (0:00:00.060) 0:01:06.525 ***** 2021-02-11T09:38:40.631 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2021-02-11T09:38:40.632 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:40.665 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:40 +0000 (0:00:00.061) 0:01:06.586 ***** 2021-02-11T09:38:40.796 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-02-11T09:38:40.797 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "size": 13355, "state": "file", "uid": 0} 2021-02-11T09:38:40.967 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2021-02-11T09:38:40.969 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "checksum": "a0ee4c31e844cf26108115f6af99f92f904eafbf", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "size": 1176, "state": "file", "uid": 0} 2021-02-11T09:38:41.009 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:41 +0000 (0:00:00.343) 0:01:06.930 ***** 2021-02-11T09:38:41.061 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-02-11T09:38:41.062 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "size": 8602, "state": "file", "uid": 0} 2021-02-11T09:38:41.327 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "size": 3605, "state": "file", "uid": 0} 2021-02-11T09:38:41.375 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:41 +0000 (0:00:00.365) 0:01:07.296 ***** 2021-02-11T09:38:41.412 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** ok: [smithi098.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib/nagios/plugins", "nrpe_group": "nagios", "nrpe_packages": ["nagios-nrpe-server", "nagios-plugins-basic"], "nrpe_service_name": "nagios-nrpe-server", "nrpe_user": "nagios"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/apt_systems.yml"], "changed": false} 2021-02-11T09:38:41.447 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:41 +0000 (0:00:00.072) 0:01:07.368 ***** 2021-02-11T09:38:41.474 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2021-02-11T09:38:41.475 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:42.105 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:41 +0000 (0:00:00.062) 0:01:07.430 ***** TASK [common : Check for epel] ************************************************* skipping: [smithi098.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} Thursday 11 February 2021 09:38:41 +0000 (0:00:00.060) 0:01:07.490 ***** TASK [common : Install nrpe without epel] ************************************** skipping: [smithi098.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi098.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi098.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi098.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3. 2021-02-11T09:38:42.107 INFO:teuthology.task.ansible.out:el7.x86_64.rpm", "skip_reason": "Conditional result was False"} Thursday 11 February 2021 09:38:41 +0000 (0:00:00.104) 0:01:07.595 ***** TASK [common : Make sure nagios nrpe service is running.] ********************** ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nagios-nrpe-server", "state": "started", "status": {"ActiveEnterTimestamp": "Thu 2021-02-11 09:36:59 UTC", "ActiveEnterTimestampMonotonic": "22408183", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "time-sync.target var-run.mount sysinit.target system.slice remote-fs.target network.target -.mount local-fs.target basic.target systemd-tmpfiles-setup.service nss-lookup.target systemd-journald.socket", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2021-02-11 09:36:59 UTC", "AssertTimestampMonotonic": "22405605", "Before": "getty@tty1.service plymouth-quit.service multi-user.target shutdown.target xdm.service", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": 2021-02-11T09:38:42.108 INFO:teuthology.task.ansible.out:"yes", "ConditionTimestamp": "Thu 2021-02-11 09:36:59 UTC", "ConditionTimestampMonotonic": "22405604", "ConfigurationDirectoryMode": "0755", "Conflicts": "nrpe.socket shutdown.target", "ControlGroup": "/system.slice/nagios-nrpe-server.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EnvironmentFile": "/etc/default/nagios-nrpe-server (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1159", "ExecMainStartTimestamp": "Thu 2021-02-11 09:36:59 UTC", "ExecMainStartTimestampMonotonic": "22408077", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; ignore_errors=no ; start_time=[Thu 2021-02-11 09:36:59 UTC] ; stop_time=[n/a] ; pid=1159 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nagios/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/nagios-nrpe-server.service", "GID": "118", "Group": "nagios", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nagios-nrpe-server.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2021-02-11 09:36:59 UTC", "InactiveExitTimestampMonotonic": "22408183", "Inv 2021-02-11T09:38:42.109 INFO:teuthology.task.ansible.out:ocationID": "638638eda22143b686c53faf04725d63", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128029", "LimitNPROCSoft": "128029", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "128029", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1159", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nagios-nrpe-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nagios/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", " 2021-02-11T09:38:42.110 INFO:teuthology.task.ansible.out:RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice -.mount sysinit.target", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2021-02-11 09:36:59 UTC", "StateChangeTimestampMonotonic": "22408183", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "109", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "User": "nagios", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2021-02-11 09:36:59 UTC", "WatchdogTimestampMonotonic": "22408181", "WatchdogUSec": "0"}} Thursday 11 February 2021 09:38:41 +0000 (0:00:00.044) 0:01:07.640 ***** TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ skipping: [smithi098.front.sepia.ceph.com] => {"changed 2021-02-11T09:38:42.112 INFO:teuthology.task.ansible.out:": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 09:38:41 +0000 (0:00:00.047) 0:01:07.687 ***** TASK [common : Get SELinux status] ********************************************* skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 09:38:41 +0000 (0:00:00.041) 0:01:07.729 ***** TASK [common : nrpe - Install SELinux dependencies] **************************** skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 09:38:41 +0000 (0:00:00.062) 0:01:07.791 ***** TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 09:38:41 +0000 (0:00:00.061) 0:01:07.852 ***** TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 09:38:41 +0000 (0:00:00.062) 0:01:07.914 ***** TASK [common : nrpe - Remove SELinux policy package] *************************** skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 09:38:42 +0000 (0:00:00.060) 0:01:07.975 ***** TASK [common : nrpe - Copy SELinux type enforcement file] ********************** skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:42.116 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:42 +0000 (0:00:00.061) 0:01:08.036 ***** 2021-02-11T09:38:42.143 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2021-02-11T09:38:42.144 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:42.176 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:42 +0000 (0:00:00.060) 0:01:08.097 ***** 2021-02-11T09:38:42.203 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2021-02-11T09:38:42.205 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:42.238 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:42 +0000 (0:00:00.061) 0:01:08.159 ***** 2021-02-11T09:38:42.260 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2021-02-11T09:38:42.261 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:42.294 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:42 +0000 (0:00:00.055) 0:01:08.215 ***** 2021-02-11T09:38:42.321 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2021-02-11T09:38:42.323 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:42.357 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:42 +0000 (0:00:00.063) 0:01:08.278 ***** 2021-02-11T09:38:42.395 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2021-02-11T09:38:42.396 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"ansible_facts": {"apt_repos": [], "ceph_packages_to_remove": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common"], "common_apt_repos": [], "common_packages": [], "nfs_service": "nfs-kernel-server", "ntp_service_name": "ntp", "packages": [], "packages_to_remove": ["multipath-tools"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "ssh"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/apt_systems.yml"], "changed": false} 2021-02-11T09:38:42.429 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:42 +0000 (0:00:00.071) 0:01:08.349 ***** 2021-02-11T09:38:42.478 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2021-02-11T09:38:42.478 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml) => {"ansible_facts": {"common_packages": ["libfcgi0ldbl", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "collectl", "blktrace", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "non_aarch64_common_packages": ["smbios-utils", "libfcgi", "sysprof"], "non_aarch64_packages_to_upgrade": ["libapache2-mod-fastcgi"], "packages_to_remove": ["openmpi-common"], "packages_to_upgrade": ["apt", "apache2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"} 2021-02-11T09:38:42.532 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:42 +0000 (0:00:00.103) 0:01:08.453 ***** 2021-02-11T09:38:42.571 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2021-02-11T09:38:42.573 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml) => {"ansible_facts": {"non_aarch64_packages": ["libgoogle-perftools4", "iozone3", "libssl1.0.0"], "non_aarch64_packages_to_upgrade": [], "packages": ["mpich", "qemu-system-x86", "python-virtualenv", "python-configobj", "python-gevent", "python-numpy", "python-matplotlib", "python-nose", "btrfs-tools", "lttng-tools", "libtool-bin", "docker.io", "python3-nose"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"} 2021-02-11T09:38:42.608 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:42 +0000 (0:00:00.075) 0:01:08.528 ***** 2021-02-11T09:38:42.614 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2021-02-11T09:38:42.616 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => {"cache_update_time": 1613036065, "cache_updated": false, "changed": false} 2021-02-11T09:38:42.656 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:42 +0000 (0:00:00.047) 0:01:08.576 ***** 2021-02-11T09:38:42.661 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2021-02-11T09:38:42.663 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-02-11T09:38:42.699 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:42 +0000 (0:00:00.043) 0:01:08.620 ***** 2021-02-11T09:38:42.829 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2021-02-11T09:38:42.831 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2021-02-11T09:38:42.863 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:42 +0000 (0:00:00.164) 0:01:08.784 ***** 2021-02-11T09:38:42.952 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2021-02-11T09:38:42.953 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => {"changed": false, "checksum": "4880e187ff9f437cc33858eb1d0fa49a06d67be4", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "size": 180, "state": "file", "uid": 0} 2021-02-11T09:38:42.990 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:42 +0000 (0:00:00.126) 0:01:08.910 ***** 2021-02-11T09:38:42.995 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2021-02-11T09:38:42.996 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "name": "ubuntu", "state": "present", "system": false} 2021-02-11T09:38:43.032 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:43 +0000 (0:00:00.042) 0:01:08.953 ***** 2021-02-11T09:38:43.058 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2021-02-11T09:38:43.059 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:43.093 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:43 +0000 (0:00:00.060) 0:01:09.013 ***** 2021-02-11T09:38:43.120 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2021-02-11T09:38:43.122 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:43.155 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:43 +0000 (0:00:00.062) 0:01:09.076 ***** 2021-02-11T09:38:43.286 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2021-02-11T09:38:43.287 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/nagios/plugins/check_mem.sh", "size": 1964, "state": "file", "uid": 0} 2021-02-11T09:38:43.320 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:43 +0000 (0:00:00.165) 0:01:09.241 ***** 2021-02-11T09:38:43.326 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2021-02-11T09:38:43.326 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/sh", "state": "present", "uid": 10101} 2021-02-11T09:38:43.363 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:43 +0000 (0:00:00.042) 0:01:09.284 ***** 2021-02-11T09:38:43.464 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-02-11T09:38:43.465 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "DAEMON_OPTS", "msg": ""} 2021-02-11T09:38:43.590 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "NRPE_OPTS", "msg": ""} 2021-02-11T09:38:43.627 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:43 +0000 (0:00:00.263) 0:01:09.547 ***** 2021-02-11T09:38:43.653 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:43.688 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:43 +0000 (0:00:00.061) 0:01:09.609 ***** 2021-02-11T09:38:43.718 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2021-02-11T09:38:43.720 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-11T09:38:43.753 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:43 +0000 (0:00:00.064) 0:01:09.673 ***** 2021-02-11T09:38:43.890 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2021-02-11T09:38:43.892 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "size": 112, "state": "file", "uid": 0} 2021-02-11T09:38:43.924 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:43 +0000 (0:00:00.171) 0:01:09.845 ***** 2021-02-11T09:38:43.931 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2021-02-11T09:38:43.932 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:43.967 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:43 +0000 (0:00:00.042) 0:01:09.887 ***** 2021-02-11T09:38:43.974 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2021-02-11T09:38:43.975 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi094"}, "changed": false} 2021-02-11T09:38:44.010 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:44 +0000 (0:00:00.043) 0:01:09.931 ***** 2021-02-11T09:38:44.288 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2021-02-11T09:38:44.290 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => {"changed": false, "checksum": "a0ee4c31e844cf26108115f6af99f92f904eafbf", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "size": 1176, "state": "file", "uid": 0} 2021-02-11T09:38:44.326 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:44 +0000 (0:00:00.315) 0:01:10.246 ***** 2021-02-11T09:38:44.530 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2021-02-11T09:38:44.531 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nagios-nrpe-server", "state": "started", "status": {"ActiveEnterTimestamp": "Thu 2021-02-11 09:37:07 UTC", "ActiveEnterTimestampMonotonic": "22798411", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "var-run.mount local-fs.target systemd-tmpfiles-setup.service -.mount systemd-journald.socket remote-fs.target system.slice basic.target nss-lookup.target network.target sysinit.target time-sync.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2021-02-11 09:37:07 UTC", "AssertTimestampMonotonic": "22796308", "Before": "xdm.service shutdown.target getty@tty1.service multi-user.target plymouth-quit.service", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2021-02-11 09:37:07 UTC", "ConditionTimestampMonotonic": "22796308", "ConfigurationDirectoryMode": "0755", "Conflicts": "nrpe.socket shutdown.target", "ControlGroup": "/system.slice/nagios-nrpe-ser 2021-02-11T09:38:44.533 INFO:teuthology.task.ansible.out:ver.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EnvironmentFile": "/etc/default/nagios-nrpe-server (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1206", "ExecMainStartTimestamp": "Thu 2021-02-11 09:37:07 UTC", "ExecMainStartTimestampMonotonic": "22798352", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; ignore_errors=no ; start_time=[Thu 2021-02-11 09:37:07 UTC] ; stop_time=[n/a] ; pid=1206 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nagios/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/nagios-nrpe-server.service", "GID": "118", "Group": "nagios", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nagios-nrpe-server.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2021-02-11 09:37:07 UTC", "InactiveExitTimestampMonotonic": "22798411", "InvocationID": "a51a42f002254b2ca9a9a6897a951a7f", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "i 2021-02-11T09:38:44.534 INFO:teuthology.task.ansible.out:nfinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128029", "LimitNPROCSoft": "128029", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "128029", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1206", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nagios-nrpe-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nagios/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target system.slice -.mount", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartUSec": "100ms", 2021-02-11T09:38:44.535 INFO:teuthology.task.ansible.out: "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2021-02-11 09:37:07 UTC", "StateChangeTimestampMonotonic": "22798411", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "109", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "User": "nagios", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2021-02-11 09:37:07 UTC", "WatchdogTimestampMonotonic": "22798409", "WatchdogUSec": "0"}} 2021-02-11T09:38:44.569 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:44 +0000 (0:00:00.243) 0:01:10.490 ***** 2021-02-11T09:38:44.596 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2021-02-11T09:38:44.597 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:44.631 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:44 +0000 (0:00:00.062) 0:01:10.552 ***** 2021-02-11T09:38:44.668 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2021-02-11T09:38:44.670 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:44.705 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:44 +0000 (0:00:00.073) 0:01:10.626 ***** 2021-02-11T09:38:44.732 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2021-02-11T09:38:44.734 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:44.768 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:44 +0000 (0:00:00.062) 0:01:10.688 ***** 2021-02-11T09:38:44.794 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2021-02-11T09:38:44.796 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:44.830 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:44 +0000 (0:00:00.062) 0:01:10.751 ***** 2021-02-11T09:38:44.837 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2021-02-11T09:38:44.837 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi094.front.sepia.ceph.com", "ansible_hostname": "smithi094", "ansible_nodename": "smithi094"}, "changed": true, "name": "smithi094"} 2021-02-11T09:38:44.879 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:44 +0000 (0:00:00.048) 0:01:10.799 ***** 2021-02-11T09:38:44.884 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2021-02-11T09:38:44.886 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:44.923 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:44 +0000 (0:00:00.044) 0:01:10.844 ***** 2021-02-11T09:38:44.954 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2021-02-11T09:38:44.956 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:44.994 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:44 +0000 (0:00:00.070) 0:01:10.914 ***** 2021-02-11T09:38:45.023 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:45.062 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:45 +0000 (0:00:00.068) 0:01:10.982 ***** 2021-02-11T09:38:45.088 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2021-02-11T09:38:45.090 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:45.123 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:45 +0000 (0:00:00.061) 0:01:11.044 ***** 2021-02-11T09:38:45.150 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2021-02-11T09:38:45.151 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:45.185 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:45 +0000 (0:00:00.061) 0:01:11.105 ***** 2021-02-11T09:38:45.201 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2021-02-11T09:38:45.202 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "checksum": "2f7bfea2455a0d9220592aa0c21ac69c312e7471", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "size": 2646, "state": "file", "uid": 0} 2021-02-11T09:38:45.236 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:45 +0000 (0:00:00.051) 0:01:11.157 ***** 2021-02-11T09:38:45.242 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2021-02-11T09:38:45.244 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:45.281 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:45 +0000 (0:00:00.045) 0:01:11.202 ***** 2021-02-11T09:38:45.307 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2021-02-11T09:38:45.309 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => {"ansible_facts": {"apt_repos": [], "ceph_packages_to_remove": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common"], "common_apt_repos": [], "common_packages": [], "nfs_service": "nfs-kernel-server", "ntp_service_name": "ntp", "packages": [], "packages_to_remove": ["multipath-tools"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "ssh"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/apt_systems.yml"], "changed": false} 2021-02-11T09:38:45.343 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:45 +0000 (0:00:00.061) 0:01:11.263 ***** 2021-02-11T09:38:45.380 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2021-02-11T09:38:45.381 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml) => {"ansible_facts": {"common_packages": ["libfcgi0ldbl", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "collectl", "blktrace", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "non_aarch64_common_packages": ["smbios-utils", "libfcgi", "sysprof"], "non_aarch64_packages_to_upgrade": ["libapache2-mod-fastcgi"], "packages_to_remove": ["openmpi-common"], "packages_to_upgrade": ["apt", "apache2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"} 2021-02-11T09:38:45.430 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:45 +0000 (0:00:00.086) 0:01:11.350 ***** 2021-02-11T09:38:45.467 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2021-02-11T09:38:45.468 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml) => {"ansible_facts": {"non_aarch64_packages": ["libgoogle-perftools4", "iozone3", "libssl1.0.0"], "non_aarch64_packages_to_upgrade": [], "packages": ["mpich", "qemu-system-x86", "python-virtualenv", "python-configobj", "python-gevent", "python-numpy", "python-matplotlib", "python-nose", "btrfs-tools", "lttng-tools", "libtool-bin", "docker.io", "python3-nose"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"} 2021-02-11T09:38:45.503 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:45 +0000 (0:00:00.073) 0:01:11.423 ***** 2021-02-11T09:38:45.540 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2021-02-11T09:38:45.542 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-02-11T09:38:45.581 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:45 +0000 (0:00:00.078) 0:01:11.501 ***** 2021-02-11T09:38:45.587 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2021-02-11T09:38:45.587 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "size": 570, "state": "file", "uid": 0} 2021-02-11T09:38:45.625 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:45 +0000 (0:00:00.044) 0:01:11.546 ***** 2021-02-11T09:38:45.715 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2021-02-11T09:38:45.716 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2021-02-11T09:38:45.750 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:45 +0000 (0:00:00.124) 0:01:11.670 ***** 2021-02-11T09:38:45.886 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2021-02-11T09:38:45.888 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "name": "ubuntu", "state": "present", "system": false} 2021-02-11T09:38:45.921 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:45 +0000 (0:00:00.171) 0:01:11.842 ***** 2021-02-11T09:38:45.948 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2021-02-11T09:38:45.949 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:45.984 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:45 +0000 (0:00:00.062) 0:01:11.904 ***** 2021-02-11T09:38:46.010 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2021-02-11T09:38:46.011 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:46.048 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:46 +0000 (0:00:00.063) 0:01:11.968 ***** 2021-02-11T09:38:46.209 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2021-02-11T09:38:46.211 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/sh", "state": "present", "uid": 10101} 2021-02-11T09:38:46.246 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:46 +0000 (0:00:00.198) 0:01:12.166 ***** 2021-02-11T09:38:46.252 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2021-02-11T09:38:46.253 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-02-11T09:38:46.291 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:46 +0000 (0:00:00.045) 0:01:12.212 ***** 2021-02-11T09:38:46.320 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2021-02-11T09:38:46.321 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:46.354 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:46 +0000 (0:00:00.062) 0:01:12.274 ***** 2021-02-11T09:38:46.382 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-02-11T09:38:46.383 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:46.416 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:46 +0000 (0:00:00.062) 0:01:12.336 ***** 2021-02-11T09:38:46.443 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-02-11T09:38:46.444 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:46.476 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:46 +0000 (0:00:00.060) 0:01:12.397 ***** 2021-02-11T09:38:46.502 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-02-11T09:38:46.504 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:46.536 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:46 +0000 (0:00:00.060) 0:01:12.457 ***** 2021-02-11T09:38:46.563 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:46.599 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:46 +0000 (0:00:00.062) 0:01:12.520 ***** 2021-02-11T09:38:46.627 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:46.627 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2021-02-11T09:38:46.627 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "size": 112, "state": "file", "uid": 0} 2021-02-11T09:38:46.663 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:46 +0000 (0:00:00.064) 0:01:12.584 ***** 2021-02-11T09:38:46.701 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:46 +0000 (0:00:00.037) 0:01:12.622 ***** 2021-02-11T09:38:46.715 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-02-11T09:38:46.716 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:46.750 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:46 +0000 (0:00:00.049) 0:01:12.671 ***** 2021-02-11T09:38:46.757 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2021-02-11T09:38:46.757 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi098"}, "changed": false} 2021-02-11T09:38:46.794 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:46 +0000 (0:00:00.043) 0:01:12.715 ***** 2021-02-11T09:38:46.800 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2021-02-11T09:38:46.801 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:46.839 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:46 +0000 (0:00:00.044) 0:01:12.760 ***** 2021-02-11T09:38:46.867 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2021-02-11T09:38:46.901 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:46 +0000 (0:00:00.061) 0:01:12.821 ***** 2021-02-11T09:38:46.929 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-02-11T09:38:46.964 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:46 +0000 (0:00:00.062) 0:01:12.884 ***** 2021-02-11T09:38:47.002 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2021-02-11T09:38:47.041 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:47 +0000 (0:00:00.077) 0:01:12.962 ***** 2021-02-11T09:38:47.068 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:47.104 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:47 +0000 (0:00:00.062) 0:01:13.024 ***** 2021-02-11T09:38:47.142 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2021-02-11T09:38:47.143 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-02-11T09:38:47.178 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:47 +0000 (0:00:00.074) 0:01:13.099 ***** 2021-02-11T09:38:47.205 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:47.239 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:47 +0000 (0:00:00.061) 0:01:13.160 ***** 2021-02-11T09:38:47.246 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2021-02-11T09:38:47.246 INFO:teuthology.task.ansible.out:changed: [smithi098.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi098.front.sepia.ceph.com", "ansible_hostname": "smithi098", "ansible_nodename": "smithi098"}, "changed": true, "name": "smithi098"} 2021-02-11T09:38:47.281 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:47 +0000 (0:00:00.041) 0:01:13.202 ***** 2021-02-11T09:38:47.286 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-11T09:38:47.288 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:47.324 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:47 +0000 (0:00:00.042) 0:01:13.244 ***** 2021-02-11T09:38:47.349 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2021-02-11T09:38:47.351 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:47.384 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:47 +0000 (0:00:00.060) 0:01:13.305 ***** 2021-02-11T09:38:47.411 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2021-02-11T09:38:47.412 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:47.450 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:47 +0000 (0:00:00.066) 0:01:13.371 ***** 2021-02-11T09:38:47.477 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2021-02-11T09:38:47.478 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:47.512 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:47 +0000 (0:00:00.061) 0:01:13.432 ***** 2021-02-11T09:38:47.538 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2021-02-11T09:38:47.540 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:47.573 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:47 +0000 (0:00:00.061) 0:01:13.494 ***** 2021-02-11T09:38:47.589 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2021-02-11T09:38:47.589 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => {"changed": false, "checksum": "2f7bfea2455a0d9220592aa0c21ac69c312e7471", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "size": 2646, "state": "file", "uid": 0} 2021-02-11T09:38:47.624 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:47 +0000 (0:00:00.050) 0:01:13.544 ***** 2021-02-11T09:38:47.630 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-02-11T09:38:47.631 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:47.668 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:47 +0000 (0:00:00.044) 0:01:13.589 ***** 2021-02-11T09:38:47.694 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-02-11T09:38:47.696 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:47.729 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:47 +0000 (0:00:00.060) 0:01:13.650 ***** 2021-02-11T09:38:47.756 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:47.791 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:47 +0000 (0:00:00.062) 0:01:13.712 ***** 2021-02-11T09:38:47.818 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2021-02-11T09:38:47.819 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:47.852 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:47 +0000 (0:00:00.059) 0:01:13.772 ***** 2021-02-11T09:38:47.878 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:47.913 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:47 +0000 (0:00:00.061) 0:01:13.833 ***** 2021-02-11T09:38:47.919 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2021-02-11T09:38:47.920 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "size": 570, "state": "file", "uid": 0} 2021-02-11T09:38:47.956 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:47 +0000 (0:00:00.042) 0:01:13.876 ***** 2021-02-11T09:38:47.961 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-02-11T09:38:47.963 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:47.999 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:47 +0000 (0:00:00.043) 0:01:13.920 ***** 2021-02-11T09:38:48.026 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2021-02-11T09:38:48.028 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:48.060 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:48 +0000 (0:00:00.061) 0:01:13.981 ***** 2021-02-11T09:38:48.087 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:48.120 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:48 +0000 (0:00:00.060) 0:01:14.041 ***** 2021-02-11T09:38:48.152 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2021-02-11T09:38:48.154 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:48.186 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:48 +0000 (0:00:00.065) 0:01:14.107 ***** 2021-02-11T09:38:48.213 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2021-02-11T09:38:48.214 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:48.247 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:48 +0000 (0:00:00.060) 0:01:14.168 ***** 2021-02-11T09:38:48.274 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2021-02-11T09:38:48.275 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:48.308 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:48 +0000 (0:00:00.061) 0:01:14.229 ***** 2021-02-11T09:38:48.335 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:48.369 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:48 +0000 (0:00:00.060) 0:01:14.290 ***** 2021-02-11T09:38:48.396 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:48.431 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:48 +0000 (0:00:00.061) 0:01:14.351 ***** 2021-02-11T09:38:48.458 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2021-02-11T09:38:48.459 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:48.491 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:48 +0000 (0:00:00.060) 0:01:14.412 ***** 2021-02-11T09:38:48.541 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2021-02-11T09:38:48.543 INFO:teuthology.task.ansible.out:changed: [smithi098.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-02-11T09:38:48.578 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:48 +0000 (0:00:00.086) 0:01:14.498 ***** 2021-02-11T09:38:48.605 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2021-02-11T09:38:48.607 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:48.640 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:48 +0000 (0:00:00.062) 0:01:14.561 ***** 2021-02-11T09:38:48.646 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2021-02-11T09:38:48.646 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "cmd": "ls -1 /etc/apt/sources.list.d/", "delta": "0:00:00.003329", "end": "2021-02-11 09:38:48.618427", "rc": 0, "start": "2021-02-11 09:38:48.615098", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-11T09:38:48.682 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:48 +0000 (0:00:00.041) 0:01:14.603 ***** 2021-02-11T09:38:48.689 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-02-11T09:38:48.691 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:48.725 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:48 +0000 (0:00:00.042) 0:01:14.645 ***** 2021-02-11T09:38:48.731 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2021-02-11T09:38:48.766 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:48 +0000 (0:00:00.040) 0:01:14.686 ***** 2021-02-11T09:38:48.771 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-02-11T09:38:48.772 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:48.808 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:48 +0000 (0:00:00.042) 0:01:14.729 ***** 2021-02-11T09:38:48.834 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-02-11T09:38:48.835 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:48.874 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:48 +0000 (0:00:00.066) 0:01:14.795 ***** 2021-02-11T09:38:48.901 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-02-11T09:38:48.903 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:48.935 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:48 +0000 (0:00:00.060) 0:01:14.856 ***** 2021-02-11T09:38:48.961 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-02-11T09:38:48.962 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:48.996 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:48 +0000 (0:00:00.060) 0:01:14.917 ***** 2021-02-11T09:38:49.033 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:49.070 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:49 +0000 (0:00:00.074) 0:01:14.991 ***** 2021-02-11T09:38:49.077 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2021-02-11T09:38:49.078 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "checksum": "ba101482124806c7f6ea9fc7b4acfda303391ad6", "dest": "/etc/apt/preferences.d/ceph.pref", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/apt/preferences.d/ceph.pref", "size": 52, "state": "file", "uid": 0} 2021-02-11T09:38:49.116 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:49 +0000 (0:00:00.045) 0:01:15.036 ***** 2021-02-11T09:38:49.124 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2021-02-11T09:38:49.125 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:49.162 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:49 +0000 (0:00:00.046) 0:01:15.082 ***** 2021-02-11T09:38:49.169 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2021-02-11T09:38:49.169 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:49.205 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:49 +0000 (0:00:00.043) 0:01:15.126 ***** 2021-02-11T09:38:49.212 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2021-02-11T09:38:49.250 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:49 +0000 (0:00:00.044) 0:01:15.170 ***** 2021-02-11T09:38:49.276 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-02-11T09:38:49.313 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:49 +0000 (0:00:00.062) 0:01:15.233 ***** 2021-02-11T09:38:49.340 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2021-02-11T09:38:49.374 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:49 +0000 (0:00:00.061) 0:01:15.295 ***** 2021-02-11T09:38:49.401 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2021-02-11T09:38:49.403 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:49.437 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:49 +0000 (0:00:00.063) 0:01:15.358 ***** 2021-02-11T09:38:49.476 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2021-02-11T09:38:49.478 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-02-11T09:38:49.487 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-02-11T09:38:49.522 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:49 +0000 (0:00:00.084) 0:01:15.443 ***** 2021-02-11T09:38:49.548 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2021-02-11T09:38:49.549 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:49.584 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:49 +0000 (0:00:00.061) 0:01:15.504 ***** 2021-02-11T09:38:49.610 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-11T09:38:49.611 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:49.647 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:49 +0000 (0:00:00.063) 0:01:15.568 ***** 2021-02-11T09:38:49.673 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2021-02-11T09:38:49.675 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:49.708 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:49 +0000 (0:00:00.060) 0:01:15.628 ***** 2021-02-11T09:38:49.734 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2021-02-11T09:38:49.735 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:49.768 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:49 +0000 (0:00:00.060) 0:01:15.689 ***** 2021-02-11T09:38:49.794 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2021-02-11T09:38:49.796 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:49.829 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:49 +0000 (0:00:00.060) 0:01:15.750 ***** 2021-02-11T09:38:49.855 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2021-02-11T09:38:49.856 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:49.890 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:49 +0000 (0:00:00.060) 0:01:15.810 ***** 2021-02-11T09:38:49.916 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-02-11T09:38:49.917 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:49.952 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:49 +0000 (0:00:00.061) 0:01:15.872 ***** 2021-02-11T09:38:49.978 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-02-11T09:38:49.979 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:50.014 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:50 +0000 (0:00:00.062) 0:01:15.934 ***** 2021-02-11T09:38:50.040 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-02-11T09:38:50.042 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:50.075 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:50 +0000 (0:00:00.061) 0:01:15.996 ***** 2021-02-11T09:38:50.101 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2021-02-11T09:38:50.103 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:50.137 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:50 +0000 (0:00:00.061) 0:01:16.057 ***** 2021-02-11T09:38:50.163 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-02-11T09:38:50.165 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:50.198 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:50 +0000 (0:00:00.061) 0:01:16.119 ***** 2021-02-11T09:38:50.225 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:50.260 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:50 +0000 (0:00:00.061) 0:01:16.181 ***** 2021-02-11T09:38:50.286 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2021-02-11T09:38:50.288 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:50.322 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:50 +0000 (0:00:00.062) 0:01:16.243 ***** 2021-02-11T09:38:50.350 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:50.388 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:50 +0000 (0:00:00.066) 0:01:16.309 ***** 2021-02-11T09:38:50.415 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2021-02-11T09:38:50.417 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:50.452 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:50 +0000 (0:00:00.063) 0:01:16.373 ***** 2021-02-11T09:38:50.479 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2021-02-11T09:38:50.481 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:50.513 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:50 +0000 (0:00:00.060) 0:01:16.433 ***** 2021-02-11T09:38:50.539 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2021-02-11T09:38:50.540 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:50.577 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:50 +0000 (0:00:00.063) 0:01:16.497 ***** 2021-02-11T09:38:50.584 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2021-02-11T09:38:50.584 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc"} 2021-02-11T09:38:50.606 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2021-02-11T09:38:50.607 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:50.642 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:50 +0000 (0:00:00.065) 0:01:16.563 ***** 2021-02-11T09:38:50.670 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2021-02-11T09:38:50.672 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:50.706 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:50 +0000 (0:00:00.063) 0:01:16.627 ***** 2021-02-11T09:38:50.735 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2021-02-11T09:38:50.737 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:50.771 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:50 +0000 (0:00:00.065) 0:01:16.692 ***** 2021-02-11T09:38:50.902 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2021-02-11T09:38:50.903 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => {"changed": false, "cmd": "ls -1 /etc/apt/sources.list.d/", "delta": "0:00:00.003326", "end": "2021-02-11 09:38:51.162331", "rc": 0, "start": "2021-02-11 09:38:51.159005", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-11T09:38:50.938 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:50 +0000 (0:00:00.166) 0:01:16.859 ***** 2021-02-11T09:38:50.965 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2021-02-11T09:38:50.999 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:50 +0000 (0:00:00.061) 0:01:16.920 ***** 2021-02-11T09:38:51.297 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2021-02-11T09:38:51.298 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => {"changed": false, "checksum": "ba101482124806c7f6ea9fc7b4acfda303391ad6", "dest": "/etc/apt/preferences.d/ceph.pref", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/apt/preferences.d/ceph.pref", "size": 52, "state": "file", "uid": 0} 2021-02-11T09:38:51.332 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:51 +0000 (0:00:00.333) 0:01:17.253 ***** 2021-02-11T09:38:51.360 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2021-02-11T09:38:51.361 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:38:51.418 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:51 +0000 (0:00:00.086) 0:01:17.339 ***** 2021-02-11T09:38:51.501 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2021-02-11T09:38:51.503 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc"} 2021-02-11T09:38:51.542 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:51 +0000 (0:00:00.123) 0:01:17.462 ***** 2021-02-11T09:38:52.399 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc"} 2021-02-11T09:38:52.442 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2021-02-11T09:38:52.443 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"cache_update_time": 1613036022, "cache_updated": false, "changed": false} 2021-02-11T09:38:52.478 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:52 +0000 (0:00:00.936) 0:01:18.398 ***** 2021-02-11T09:38:52.505 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2021-02-11T09:38:52.543 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:52 +0000 (0:00:00.065) 0:01:18.463 ***** 2021-02-11T09:38:52.705 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2021-02-11T09:38:52.707 INFO:teuthology.task.ansible.out:[WARNING]: Consider using 'become', 'become_method', and 'become_user' rather than running sudo changed: [smithi094.front.sepia.ceph.com] => {"changed": true, "cmd": ["sudo", "apt-get", "clean"], "delta": "0:00:00.036964", "end": "2021-02-11 09:38:52.690037", "rc": 0, "start": "2021-02-11 09:38:52.653073", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-11T09:38:52.742 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:52 +0000 (0:00:00.198) 0:01:18.662 ***** 2021-02-11T09:38:53.303 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2021-02-11T09:38:53.304 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc"} 2021-02-11T09:38:53.342 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:53 +0000 (0:00:00.599) 0:01:19.262 ***** 2021-02-11T09:38:54.221 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2021-02-11T09:38:54.223 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => {"cache_update_time": 1613036065, "cache_updated": false, "changed": false} 2021-02-11T09:38:54.259 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:54 +0000 (0:00:00.917) 0:01:20.180 ***** 2021-02-11T09:38:54.285 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2021-02-11T09:38:54.319 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:54 +0000 (0:00:00.060) 0:01:20.240 ***** 2021-02-11T09:38:54.491 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2021-02-11T09:38:54.492 INFO:teuthology.task.ansible.out:changed: [smithi098.front.sepia.ceph.com] => {"changed": true, "cmd": ["sudo", "apt-get", "clean"], "delta": "0:00:00.035121", "end": "2021-02-11 09:38:54.742867", "rc": 0, "start": "2021-02-11 09:38:54.707746", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-11T09:38:54.526 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:54 +0000 (0:00:00.207) 0:01:20.447 ***** 2021-02-11T09:38:55.530 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2021-02-11T09:38:55.531 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"cache_update_time": 1613036022, "cache_updated": false, "changed": false} 2021-02-11T09:38:55.567 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:55 +0000 (0:00:01.040) 0:01:21.488 ***** 2021-02-11T09:38:57.423 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => {"cache_update_time": 1613036065, "cache_updated": false, "changed": false} 2021-02-11T09:38:57.463 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:38:57 +0000 (0:00:01.895) 0:01:23.384 ***** 2021-02-11T09:39:04.402 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2021-02-11T09:39:04.403 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => {"changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following packages will be REMOVED:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 0 newly installed, 6 to remove and 19 not upgraded.\nAfter this operation, 47.6 MB disk space will be freed.\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137610 files and directorie 2021-02-11T09:39:04.405 INFO:teuthology.task.ansible.out:s currently installed.)\r\nRemoving qemu-utils (1:2.11+dfsg-1ubuntu7.34) ...\r\nRemoving qemu-system-x86 (1:2.11+dfsg-1ubuntu7.34) ...\r\nRemoving qemu-system-common (1:2.11+dfsg-1ubuntu7.34) ...\r\nRemoving qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.34) ...\r\nRemoving librbd1 (12.2.13-0ubuntu0.18.04.6) ...\r\nRemoving librados2 (12.2.13-0ubuntu0.18.04.6) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1.4) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following packages will be REMOVED:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 0 newly installed, 6 to remove and 19 not upgraded.", "After this operation, 47.6 MB disk space will be freed.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Read 2021-02-11T09:39:04.406 INFO:teuthology.task.ansible.out:ing database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137610 files and directories currently installed.)", "Removing qemu-utils (1:2.11+dfsg-1ubuntu7.34) ...", "Removing qemu-system-x86 (1:2.11+dfsg-1ubuntu7.34) ...", "Removing qemu-system-common (1:2.11+dfsg-1ubuntu7.34) ...", "Removing qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.34) ...", "Removing librbd1 (12.2.13-0ubuntu0.18.04.6) ...", "Removing librados2 (12.2.13-0ubuntu0.18.04.6) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.4) ..."]} 2021-02-11T09:39:04.440 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:04 +0000 (0:00:06.976) 0:01:30.361 ***** 2021-02-11T09:39:05.332 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-02-11T09:39:05.334 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"cache_update_time": 1613036022, "cache_updated": false, "changed": false} 2021-02-11T09:39:05.367 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:05 +0000 (0:00:00.926) 0:01:31.287 ***** 2021-02-11T09:39:05.403 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2021-02-11T09:39:05.405 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:05.439 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:05 +0000 (0:00:00.072) 0:01:31.360 ***** 2021-02-11T09:39:06.078 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2021-02-11T09:39:06.078 INFO:teuthology.task.ansible.out:changed: [smithi098.front.sepia.ceph.com] => {"changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following packages will be REMOVED:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 0 newly installed, 6 to remove and 19 not upgraded.\nAfter this operation, 47.6 MB disk space will be freed.\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137610 files and directorie 2021-02-11T09:39:06.079 INFO:teuthology.task.ansible.out:s currently installed.)\r\nRemoving qemu-utils (1:2.11+dfsg-1ubuntu7.34) ...\r\nRemoving qemu-system-x86 (1:2.11+dfsg-1ubuntu7.34) ...\r\nRemoving qemu-system-common (1:2.11+dfsg-1ubuntu7.34) ...\r\nRemoving qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.34) ...\r\nRemoving librbd1 (12.2.13-0ubuntu0.18.04.6) ...\r\nRemoving librados2 (12.2.13-0ubuntu0.18.04.6) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1.4) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following packages will be REMOVED:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 0 newly installed, 6 to remove and 19 not upgraded.", "After this operation, 47.6 MB disk space will be freed.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Read 2021-02-11T09:39:06.081 INFO:teuthology.task.ansible.out:ing database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137610 files and directories currently installed.)", "Removing qemu-utils (1:2.11+dfsg-1ubuntu7.34) ...", "Removing qemu-system-x86 (1:2.11+dfsg-1ubuntu7.34) ...", "Removing qemu-system-common (1:2.11+dfsg-1ubuntu7.34) ...", "Removing qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.34) ...", "Removing librbd1 (12.2.13-0ubuntu0.18.04.6) ...", "Removing librados2 (12.2.13-0ubuntu0.18.04.6) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.4) ..."]} 2021-02-11T09:39:06.115 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:06 +0000 (0:00:00.675) 0:01:32.035 ***** 2021-02-11T09:39:07.044 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-02-11T09:39:07.046 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => {"cache_update_time": 1613036065, "cache_updated": false, "changed": false} 2021-02-11T09:39:07.077 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:07 +0000 (0:00:00.962) 0:01:32.998 ***** 2021-02-11T09:39:07.103 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2021-02-11T09:39:07.104 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:07.138 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:07 +0000 (0:00:00.060) 0:01:33.059 ***** 2021-02-11T09:39:14.131 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-02-11T09:39:14.132 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => {"cache_update_time": 1613036022, "cache_updated": false, "changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\ngit is already the newest version (1:2.17.1-1ubuntu0.7).\nThe following additional packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils\nSuggested packages:\n samba vde2 sgabios ovmf debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 6 newly installed, 0 to remove and 19 not upgraded.\nNeed to get 10.4 MB of archives.\nAfter this operation, 47.6 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.6 [2724 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.6 [925 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.35 [40.5 kB]\nGet:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.35 [672 kB]\nGet:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.35 [5194 kB]\nGet:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.35 [868 kB]\nFetched 10.4 MB in 1s (9870 kB/s)\nSelecting previously unselected package librados2.\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database .. 2021-02-11T09:39:14.134 INFO:teuthology.task.ansible.out:. 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137468 files and directories currently installed.)\r\nPreparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.6_amd64.deb ...\r\nUnpacking librados2 (12.2.13-0ubuntu0.18.04.6) ...\r\nSelecting previously unselected package librbd1.\r\nPreparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.6_amd64.deb ...\r\nUnpacking librbd1 (12.2.13-0ubuntu0.18.04.6) ...\r\nSelecting previously unselected package qemu-block-extra:amd64.\r\nPreparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.35_amd64.deb ...\r\nUnpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.35) ...\r\nSelecting previously unselected package qemu-system-common.\r\nPreparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.35_amd64.deb ...\r\nUnpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.35) ...\r\nSelecting previously unselected package qemu-system-x86.\r\nPreparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.35_amd64.deb ...\r\nUnpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.35) ...\r\nSelecting previously unselected package qemu-utils.\r\nPreparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.35_amd64.deb ...\r\nUnpacking qemu-utils (1:2.11+dfsg-1ubuntu7.35) ...\r\nSetting up librados2 (12.2.13-0ubuntu0.18.04.6) ...\r\nSetting up librbd1 (12.2.13-0ubuntu0.18.04.6) ...\r\nSetting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.35) ...\r\nSetting up qemu-utils (1:2.11+dfsg-1ubuntu7.35) ...\r\nSetting up qemu-system-common (1:2.11+dfsg-1ubuntu7.35) ...\r\nSetting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.35) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubunt 2021-02-11T09:39:14.135 INFO:teuthology.task.ansible.out:u1.4) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "git is already the newest version (1:2.17.1-1ubuntu0.7).", "The following additional packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils", "Suggested packages:", " samba vde2 sgabios ovmf debootstrap", "The following NEW packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 6 newly installed, 0 to remove and 19 not upgraded.", "Need to get 10.4 MB of archives.", "After this operation, 47.6 MB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.6 [2724 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.6 [925 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.35 [40.5 kB]", "Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.35 [672 kB]", "Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.35 [5194 kB]", "Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.35 [868 kB]", "Fetched 10.4 MB in 1s (9870 kB/s)", "Selecting previously unselected package librados2.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(R 2021-02-11T09:39:14.136 INFO:teuthology.task.ansible.out:eading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137468 files and directories currently installed.)", "Preparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.6_amd64.deb ...", "Unpacking librados2 (12.2.13-0ubuntu0.18.04.6) ...", "Selecting previously unselected package librbd1.", "Preparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.6_amd64.deb ...", "Unpacking librbd1 (12.2.13-0ubuntu0.18.04.6) ...", "Selecting previously unselected package qemu-block-extra:amd64.", "Preparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.35_amd64.deb ...", "Unpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.35) ...", "Selecting previously unselected package qemu-system-common.", "Preparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.35_amd64.deb ...", "Unpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.35) ...", "Selecting previously unselected package qemu-system-x86.", "Preparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.35_amd64.deb ...", "Unpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.35) ...", "Selecting previously unselected package qemu-utils.", "Preparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.35_amd64.deb ...", "Unpacking qemu-utils (1:2.11+dfsg-1ubuntu7.35) ...", "Setting up librados2 (12.2.13-0ubuntu0.18.04.6) ...", "Setting up librbd1 (12.2.13-0ubuntu0.18.04.6) ...", "Setting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.35) ...", "Setting up qemu-utils (1:2.11+dfsg-1ubuntu7.35) ...", "Setting up qemu-system-common (1:2.11+dfsg-1ubuntu7.35) ...", "Setting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.35) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.4) ..."]} 2021-02-11T09:39:14.189 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:14 +0000 (0:00:07.051) 0:01:40.110 ***** 2021-02-11T09:39:15.074 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2021-02-11T09:39:15.076 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"cache_update_time": 1613036022, "cache_updated": false, "changed": false} 2021-02-11T09:39:15.108 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:15 +0000 (0:00:00.918) 0:01:41.028 ***** 2021-02-11T09:39:15.226 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2021-02-11T09:39:15.228 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "cmd": ["perl", "-pe", "if (m{^([^#]\\S*\\s+/\\s+\\S+\\s+)(\\S+)(\\s+.*)$}) { $_=\"$1$2,user_xattr$3\\n\" unless $2=~m{(^|,)user_xattr(,|$)}; }", "-i.bak", "/etc/fstab"], "rc": 0, "stdout": "skipped, since /etc/fstab.bak exists", "stdout_lines": ["skipped, since /etc/fstab.bak exists"]} 2021-02-11T09:39:15.264 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:15 +0000 (0:00:00.156) 0:01:41.185 ***** 2021-02-11T09:39:15.290 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2021-02-11T09:39:15.292 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:15.325 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:15 +0000 (0:00:00.060) 0:01:41.245 ***** 2021-02-11T09:39:15.454 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2021-02-11T09:39:15.455 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "fuse", "state": "present", "system": false} 2021-02-11T09:39:15.489 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:15 +0000 (0:00:00.164) 0:01:41.410 ***** 2021-02-11T09:39:15.810 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2021-02-11T09:39:15.810 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "checksum": "9edff47031d0f14a9b8c813b44eb4a9296d0439f", "dest": "/etc/fuse.conf", "gid": 10102, "group": "fuse", "mode": "0644", "owner": "root", "path": "/etc/fuse.conf", "size": 366, "state": "file", "uid": 0} 2021-02-11T09:39:15.846 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:15 +0000 (0:00:00.356) 0:01:41.766 ***** 2021-02-11T09:39:15.997 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2021-02-11T09:39:15.998 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1238, "groups": "fuse,kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-02-11T09:39:16.032 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:16 +0000 (0:00:00.186) 0:01:41.953 ***** 2021-02-11T09:39:16.105 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-02-11T09:39:16.106 INFO:teuthology.task.ansible.out:changed: [smithi098.front.sepia.ceph.com] => {"cache_update_time": 1613036065, "cache_updated": false, "changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\ngit is already the newest version (1:2.17.1-1ubuntu0.7).\nThe following additional packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils\nSuggested packages:\n samba vde2 sgabios ovmf debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 6 newly installed, 0 to remove and 19 not upgraded.\nNeed to get 10.4 MB of archives.\nAfter this operation, 47.6 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.6 [2724 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.6 [925 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.35 [40.5 kB]\nGet:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.35 [672 kB]\nGet:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.35 [5194 kB]\nGet:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.35 [868 kB]\nFetched 10.4 MB in 1s (9440 kB/s)\nSelecting previously unselected package librados2.\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database .. 2021-02-11T09:39:16.107 INFO:teuthology.task.ansible.out:. 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137468 files and directories currently installed.)\r\nPreparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.6_amd64.deb ...\r\nUnpacking librados2 (12.2.13-0ubuntu0.18.04.6) ...\r\nSelecting previously unselected package librbd1.\r\nPreparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.6_amd64.deb ...\r\nUnpacking librbd1 (12.2.13-0ubuntu0.18.04.6) ...\r\nSelecting previously unselected package qemu-block-extra:amd64.\r\nPreparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.35_amd64.deb ...\r\nUnpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.35) ...\r\nSelecting previously unselected package qemu-system-common.\r\nPreparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.35_amd64.deb ...\r\nUnpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.35) ...\r\nSelecting previously unselected package qemu-system-x86.\r\nPreparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.35_amd64.deb ...\r\nUnpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.35) ...\r\nSelecting previously unselected package qemu-utils.\r\nPreparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.35_amd64.deb ...\r\nUnpacking qemu-utils (1:2.11+dfsg-1ubuntu7.35) ...\r\nSetting up librados2 (12.2.13-0ubuntu0.18.04.6) ...\r\nSetting up librbd1 (12.2.13-0ubuntu0.18.04.6) ...\r\nSetting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.35) ...\r\nSetting up qemu-utils (1:2.11+dfsg-1ubuntu7.35) ...\r\nSetting up qemu-system-common (1:2.11+dfsg-1ubuntu7.35) ...\r\nSetting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.35) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubunt 2021-02-11T09:39:16.108 INFO:teuthology.task.ansible.out:u1.4) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "git is already the newest version (1:2.17.1-1ubuntu0.7).", "The following additional packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils", "Suggested packages:", " samba vde2 sgabios ovmf debootstrap", "The following NEW packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 6 newly installed, 0 to remove and 19 not upgraded.", "Need to get 10.4 MB of archives.", "After this operation, 47.6 MB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.6 [2724 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.6 [925 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.35 [40.5 kB]", "Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.35 [672 kB]", "Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.35 [5194 kB]", "Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.35 [868 kB]", "Fetched 10.4 MB in 1s (9440 kB/s)", "Selecting previously unselected package librados2.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(R 2021-02-11T09:39:16.110 INFO:teuthology.task.ansible.out:eading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137468 files and directories currently installed.)", "Preparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.6_amd64.deb ...", "Unpacking librados2 (12.2.13-0ubuntu0.18.04.6) ...", "Selecting previously unselected package librbd1.", "Preparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.6_amd64.deb ...", "Unpacking librbd1 (12.2.13-0ubuntu0.18.04.6) ...", "Selecting previously unselected package qemu-block-extra:amd64.", "Preparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.35_amd64.deb ...", "Unpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.35) ...", "Selecting previously unselected package qemu-system-common.", "Preparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.35_amd64.deb ...", "Unpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.35) ...", "Selecting previously unselected package qemu-system-x86.", "Preparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.35_amd64.deb ...", "Unpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.35) ...", "Selecting previously unselected package qemu-utils.", "Preparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.35_amd64.deb ...", "Unpacking qemu-utils (1:2.11+dfsg-1ubuntu7.35) ...", "Setting up librados2 (12.2.13-0ubuntu0.18.04.6) ...", "Setting up librbd1 (12.2.13-0ubuntu0.18.04.6) ...", "Setting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.35) ...", "Setting up qemu-utils (1:2.11+dfsg-1ubuntu7.35) ...", "Setting up qemu-system-common (1:2.11+dfsg-1ubuntu7.35) ...", "Setting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.35) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.4) ..."]} 2021-02-11T09:39:16.143 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:16 +0000 (0:00:00.110) 0:01:42.064 ***** 2021-02-11T09:39:16.170 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2021-02-11T09:39:16.172 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-11T09:39:16.206 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:16 +0000 (0:00:00.063) 0:01:42.127 ***** 2021-02-11T09:39:16.514 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2021-02-11T09:39:16.515 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => {"changed": true, "name": "apache2", "state": "stopped", "status": {"ActiveEnterTimestamp": "Thu 2021-02-11 09:36:59 UTC", "ActiveEnterTimestampMonotonic": "22869649", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-tmpfiles-setup.service sysinit.target remote-fs.target system.slice basic.target -.mount nss-lookup.target network.target systemd-journald.socket", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2021-02-11 09:36:59 UTC", "AssertTimestampMonotonic": "22403118", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2021-02-11 09:36:59 UTC", "ConditionTimestampMonotonic": "22403117", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/apache2.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "The Apache HTTP Server", "DevicePolicy": "auto", " 2021-02-11T09:39:16.516 INFO:teuthology.task.ansible.out:DropInPaths": "/lib/systemd/system/apache2.service.d/apache2-systemd.conf", "DynamicUser": "no", "Environment": "APACHE_STARTED_BY_SYSTEMD=true", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1227", "ExecMainStartTimestamp": "Thu 2021-02-11 09:36:59 UTC", "ExecMainStartTimestampMonotonic": "22869621", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/apache2.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "apache2.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2021-02-11 09:36:59 UTC", "InactiveExitTimestampMonotonic": "22405498", "InvocationID": "07743006d3b145449ee2a4b05c004166", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESof 2021-02-11T09:39:16.517 INFO:teuthology.task.ansible.out:t": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128029", "LimitNPROCSoft": "128029", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "128029", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1227", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "apache2.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target -.mount", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIG 2021-02-11T09:39:16.519 INFO:teuthology.task.ansible.out:HUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2021-02-11 09:36:59 UTC", "StateChangeTimestampMonotonic": "22869649", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "55", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2021-02-11 09:36:59 UTC", "WatchdogTimestampMonotonic": "22869647", "WatchdogUSec": "0"}} 2021-02-11T09:39:16.550 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:16 +0000 (0:00:00.343) 0:01:42.471 ***** 2021-02-11T09:39:16.577 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-02-11T09:39:16.578 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:16.608 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:16 +0000 (0:00:00.058) 0:01:42.529 ***** 2021-02-11T09:39:16.626 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-02-11T09:39:16.627 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:16.658 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:16 +0000 (0:00:00.049) 0:01:42.579 ***** 2021-02-11T09:39:16.684 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-02-11T09:39:16.686 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:16.717 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:16 +0000 (0:00:00.059) 0:01:42.638 ***** 2021-02-11T09:39:16.743 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-02-11T09:39:16.745 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:16.775 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:16 +0000 (0:00:00.058) 0:01:42.696 ***** 2021-02-11T09:39:16.801 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-02-11T09:39:16.802 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:16.834 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:16 +0000 (0:00:00.059) 0:01:42.755 ***** 2021-02-11T09:39:16.864 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-02-11T09:39:16.865 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:16.898 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:16 +0000 (0:00:00.063) 0:01:42.818 ***** 2021-02-11T09:39:16.924 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-02-11T09:39:16.925 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:16.958 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:16 +0000 (0:00:00.060) 0:01:42.878 ***** 2021-02-11T09:39:16.984 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:17.018 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:17 +0000 (0:00:00.059) 0:01:42.938 ***** 2021-02-11T09:39:17.023 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2021-02-11T09:39:17.025 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => {"cache_update_time": 1613036065, "cache_updated": false, "changed": false} 2021-02-11T09:39:17.057 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:17 +0000 (0:00:00.039) 0:01:42.978 ***** 2021-02-11T09:39:17.061 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-11T09:39:17.063 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:17.097 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:17 +0000 (0:00:00.040) 0:01:43.018 ***** 2021-02-11T09:39:17.123 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-02-11T09:39:17.124 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:17.156 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:17 +0000 (0:00:00.059) 0:01:43.077 ***** 2021-02-11T09:39:17.182 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-02-11T09:39:17.183 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** ok: [smithi098.front.sepia.ceph.com] => {"changed": false, "cmd": ["perl", "-pe", "if (m{^([^#]\\S*\\s+/\\s+\\S+\\s+)(\\S+)(\\s+.*)$}) { $_=\"$1$2,user_xattr$3\\n\" unless $2=~m{(^|,)user_xattr(,|$)}; }", "-i.bak", "/etc/fstab"], "rc": 0, "stdout": "skipped, since /etc/fstab.bak exists", "stdout_lines": ["skipped, since /etc/fstab.bak exists"]} 2021-02-11T09:39:17.215 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:17 +0000 (0:00:00.059) 0:01:43.136 ***** 2021-02-11T09:39:17.245 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:17 +0000 (0:00:00.029) 0:01:43.166 ***** 2021-02-11T09:39:17.249 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-02-11T09:39:17.254 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:17.284 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:17 +0000 (0:00:00.039) 0:01:43.205 ***** 2021-02-11T09:39:17.289 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2021-02-11T09:39:17.291 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:17.323 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:17 +0000 (0:00:00.038) 0:01:43.243 ***** 2021-02-11T09:39:17.329 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-02-11T09:39:17.330 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:17.365 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:17 +0000 (0:00:00.041) 0:01:43.285 ***** 2021-02-11T09:39:17.391 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-02-11T09:39:17.393 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:17.426 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:17 +0000 (0:00:00.061) 0:01:43.347 ***** 2021-02-11T09:39:17.442 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2021-02-11T09:39:17.444 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "fuse", "state": "present", "system": false} 2021-02-11T09:39:17.476 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:17 +0000 (0:00:00.050) 0:01:43.397 ***** 2021-02-11T09:39:17.481 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-02-11T09:39:17.481 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:17.519 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:17 +0000 (0:00:00.042) 0:01:43.440 ***** 2021-02-11T09:39:17.545 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-02-11T09:39:17.547 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:17.580 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:17 +0000 (0:00:00.060) 0:01:43.501 ***** 2021-02-11T09:39:17.607 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-02-11T09:39:17.608 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:17.641 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:17 +0000 (0:00:00.060) 0:01:43.562 ***** 2021-02-11T09:39:17.667 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-02-11T09:39:17.669 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:17.701 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:17 +0000 (0:00:00.060) 0:01:43.622 ***** 2021-02-11T09:39:17.728 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-02-11T09:39:17.729 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:17.763 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:17 +0000 (0:00:00.061) 0:01:43.684 ***** 2021-02-11T09:39:17.790 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-02-11T09:39:17.791 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:17.824 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:17 +0000 (0:00:00.061) 0:01:43.745 ***** 2021-02-11T09:39:17.830 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2021-02-11T09:39:17.830 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => {"changed": false, "checksum": "9edff47031d0f14a9b8c813b44eb4a9296d0439f", "dest": "/etc/fuse.conf", "gid": 10102, "group": "fuse", "mode": "0644", "owner": "root", "path": "/etc/fuse.conf", "size": 366, "state": "file", "uid": 0} 2021-02-11T09:39:17.866 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:17 +0000 (0:00:00.041) 0:01:43.787 ***** 2021-02-11T09:39:17.871 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-02-11T09:39:17.872 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:17.907 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:17 +0000 (0:00:00.041) 0:01:43.828 ***** 2021-02-11T09:39:17.933 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-02-11T09:39:17.935 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:17.966 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:17 +0000 (0:00:00.058) 0:01:43.886 ***** 2021-02-11T09:39:17.991 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-02-11T09:39:17.992 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:18.024 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:18 +0000 (0:00:00.058) 0:01:43.944 ***** 2021-02-11T09:39:18.030 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2021-02-11T09:39:18.030 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1238, "groups": "fuse,kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-02-11T09:39:18.063 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:18 +0000 (0:00:00.039) 0:01:43.984 ***** 2021-02-11T09:39:18.068 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-02-11T09:39:18.069 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:18.103 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:18 +0000 (0:00:00.039) 0:01:44.023 ***** 2021-02-11T09:39:18.129 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-02-11T09:39:18.130 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:18.162 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:18 +0000 (0:00:00.059) 0:01:44.083 ***** 2021-02-11T09:39:18.188 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2021-02-11T09:39:18.189 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:18.225 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:18 +0000 (0:00:00.062) 0:01:44.146 ***** 2021-02-11T09:39:18.231 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2021-02-11T09:39:18.232 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-11T09:39:18.270 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:18 +0000 (0:00:00.044) 0:01:44.190 ***** 2021-02-11T09:39:18.359 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2021-02-11T09:39:18.360 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "path": "/etc/ceph", "state": "absent"} 2021-02-11T09:39:18.394 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:18 +0000 (0:00:00.124) 0:01:44.315 ***** 2021-02-11T09:39:18.556 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2021-02-11T09:39:18.557 INFO:teuthology.task.ansible.out:changed: [smithi098.front.sepia.ceph.com] => {"changed": true, "name": "apache2", "state": "stopped", "status": {"ActiveEnterTimestamp": "Thu 2021-02-11 09:37:07 UTC", "ActiveEnterTimestampMonotonic": "23206055", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-journald.socket basic.target sysinit.target nss-lookup.target remote-fs.target systemd-tmpfiles-setup.service system.slice -.mount network.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2021-02-11 09:37:07 UTC", "AssertTimestampMonotonic": "22790794", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2021-02-11 09:37:07 UTC", "ConditionTimestampMonotonic": "22790794", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/apache2.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "The Apache HTTP Server", "DevicePolicy": "auto", " 2021-02-11T09:39:18.559 INFO:teuthology.task.ansible.out:DropInPaths": "/lib/systemd/system/apache2.service.d/apache2-systemd.conf", "DynamicUser": "no", "Environment": "APACHE_STARTED_BY_SYSTEMD=true", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1287", "ExecMainStartTimestamp": "Thu 2021-02-11 09:37:07 UTC", "ExecMainStartTimestampMonotonic": "23206026", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/apache2.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "apache2.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2021-02-11 09:37:07 UTC", "InactiveExitTimestampMonotonic": "22793342", "InvocationID": "0f5a21db75c0401781ad2e9773251174", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESof 2021-02-11T09:39:18.560 INFO:teuthology.task.ansible.out:t": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128029", "LimitNPROCSoft": "128029", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "128029", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1287", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "apache2.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount system.slice sysinit.target", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIG 2021-02-11T09:39:18.561 INFO:teuthology.task.ansible.out:HUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2021-02-11 09:37:07 UTC", "StateChangeTimestampMonotonic": "23206055", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "55", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2021-02-11 09:37:07 UTC", "WatchdogTimestampMonotonic": "23206053", "WatchdogUSec": "0"}} 2021-02-11T09:39:18.594 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:18 +0000 (0:00:00.199) 0:01:44.515 ***** 2021-02-11T09:39:18.620 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-02-11T09:39:18.621 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:18.655 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:18 +0000 (0:00:00.060) 0:01:44.575 ***** 2021-02-11T09:39:18.681 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-02-11T09:39:18.682 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:18.717 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:18 +0000 (0:00:00.061) 0:01:44.637 ***** 2021-02-11T09:39:18.722 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-02-11T09:39:18.723 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "size": 578, "state": "file", "uid": 0} 2021-02-11T09:39:18.759 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:18 +0000 (0:00:00.042) 0:01:44.680 ***** 2021-02-11T09:39:18.765 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-02-11T09:39:18.767 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:18.802 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:18 +0000 (0:00:00.042) 0:01:44.722 ***** 2021-02-11T09:39:18.807 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-02-11T09:39:18.809 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:18.844 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:18 +0000 (0:00:00.042) 0:01:44.764 ***** 2021-02-11T09:39:18.850 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-02-11T09:39:18.850 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:18.887 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:18 +0000 (0:00:00.042) 0:01:44.807 ***** 2021-02-11T09:39:18.912 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:18.946 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:18 +0000 (0:00:00.059) 0:01:44.867 ***** 2021-02-11T09:39:18.972 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-02-11T09:39:18.973 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:19.006 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:19 +0000 (0:00:00.059) 0:01:44.927 ***** 2021-02-11T09:39:19.032 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-02-11T09:39:19.034 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:19.071 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:19 +0000 (0:00:00.064) 0:01:44.992 ***** 2021-02-11T09:39:19.097 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-02-11T09:39:19.098 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:19.132 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:19 +0000 (0:00:00.061) 0:01:45.053 ***** 2021-02-11T09:39:19.152 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2021-02-11T09:39:19.153 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "checksum": "d5677bbd13857966eb0fc86f66195b18a2b98e20", "dest": "/etc/cron.weekly/kernel-clean", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cron.weekly/kernel-clean", "size": 752, "state": "file", "uid": 0} 2021-02-11T09:39:19.186 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:19 +0000 (0:00:00.053) 0:01:45.106 ***** 2021-02-11T09:39:19.191 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-11T09:39:19.193 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:19.227 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:19 +0000 (0:00:00.040) 0:01:45.147 ***** 2021-02-11T09:39:19.252 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-02-11T09:39:19.253 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:19.286 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:19 +0000 (0:00:00.059) 0:01:45.207 ***** 2021-02-11T09:39:19.314 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-02-11T09:39:19.316 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:19.348 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:19 +0000 (0:00:00.061) 0:01:45.269 ***** 2021-02-11T09:39:19.375 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-02-11T09:39:19.376 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:19.409 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:19 +0000 (0:00:00.061) 0:01:45.330 ***** 2021-02-11T09:39:19.435 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-02-11T09:39:19.437 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:19.469 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:19 +0000 (0:00:00.060) 0:01:45.390 ***** 2021-02-11T09:39:19.496 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-02-11T09:39:19.497 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "checksum": "22e9a1446f43d0440219b9af14a66dd77fb0c2eb", "dest": "/etc/grub.d/02_force_timeout", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/grub.d/02_force_timeout", "size": 115, "state": "file", "uid": 0} 2021-02-11T09:39:19.531 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:19 +0000 (0:00:00.061) 0:01:45.451 ***** 2021-02-11T09:39:19.563 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:19 +0000 (0:00:00.032) 0:01:45.484 ***** 2021-02-11T09:39:19.590 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-02-11T09:39:19.591 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:19.625 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:19 +0000 (0:00:00.061) 0:01:45.545 ***** 2021-02-11T09:39:19.651 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-02-11T09:39:19.652 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:19.686 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:19 +0000 (0:00:00.061) 0:01:45.607 ***** 2021-02-11T09:39:19.712 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-02-11T09:39:19.714 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:19.748 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:19 +0000 (0:00:00.061) 0:01:45.668 ***** 2021-02-11T09:39:19.774 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-02-11T09:39:19.775 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:19.811 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:19 +0000 (0:00:00.063) 0:01:45.732 ***** 2021-02-11T09:39:19.838 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2021-02-11T09:39:19.839 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "checksum": "6449466d762bced58744df80432d189e35329803", "dest": "/etc/modules", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/modules", "size": 315, "state": "file", "uid": 0} TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:19.873 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:19 +0000 (0:00:00.061) 0:01:45.793 ***** 2021-02-11T09:39:19.907 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:19 +0000 (0:00:00.033) 0:01:45.827 ***** 2021-02-11T09:39:19.932 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-02-11T09:39:19.933 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:19.967 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:19 +0000 (0:00:00.060) 0:01:45.888 ***** 2021-02-11T09:39:19.993 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-02-11T09:39:19.995 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:20.027 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:20 +0000 (0:00:00.059) 0:01:45.947 ***** 2021-02-11T09:39:20.032 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2021-02-11T09:39:20.032 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-11T09:39:20.068 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:20 +0000 (0:00:00.041) 0:01:45.989 ***** 2021-02-11T09:39:20.073 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-02-11T09:39:20.075 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:20.110 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:20 +0000 (0:00:00.041) 0:01:46.030 ***** 2021-02-11T09:39:20.114 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2021-02-11T09:39:20.116 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:20.150 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:20 +0000 (0:00:00.040) 0:01:46.071 ***** 2021-02-11T09:39:20.155 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-02-11T09:39:20.157 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:20.191 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:20 +0000 (0:00:00.041) 0:01:46.112 ***** 2021-02-11T09:39:20.197 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2021-02-11T09:39:20.199 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:20.232 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:20 +0000 (0:00:00.041) 0:01:46.153 ***** 2021-02-11T09:39:20.239 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-02-11T09:39:20.240 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:20.274 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:20 +0000 (0:00:00.041) 0:01:46.195 ***** 2021-02-11T09:39:20.280 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2021-02-11T09:39:20.281 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:20.315 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:20 +0000 (0:00:00.040) 0:01:46.235 ***** 2021-02-11T09:39:20.319 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-02-11T09:39:20.321 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:20.353 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:20 +0000 (0:00:00.038) 0:01:46.274 ***** 2021-02-11T09:39:20.358 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2021-02-11T09:39:20.358 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:20.394 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:20 +0000 (0:00:00.040) 0:01:46.314 ***** 2021-02-11T09:39:20.399 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2021-02-11T09:39:20.401 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:20.433 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:20 +0000 (0:00:00.039) 0:01:46.354 ***** 2021-02-11T09:39:20.439 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2021-02-11T09:39:20.440 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:20.474 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:20 +0000 (0:00:00.041) 0:01:46.395 ***** 2021-02-11T09:39:20.500 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-11T09:39:20.502 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-02-11T09:39:20.534 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:20 +0000 (0:00:00.059) 0:01:46.454 ***** 2021-02-11T09:39:20.560 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2021-02-11T09:39:20.561 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => {"changed": false, "path": "/etc/ceph", "state": "absent"} 2021-02-11T09:39:20.597 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:20 +0000 (0:00:00.063) 0:01:46.518 ***** 2021-02-11T09:39:20.604 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2021-02-11T09:39:20.604 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item=loop1) => {"ansible_loop_var": "item", "changed": false, "item": "loop1", "skip_reason": "Conditional result was False"} 2021-02-11T09:39:20.605 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item=loop6) => {"ansible_loop_var": "item", "changed": false, "item": "loop6", "skip_reason": "Conditional result was False"} 2021-02-11T09:39:20.606 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item=loop4) => {"ansible_loop_var": "item", "changed": false, "item": "loop4", "skip_reason": "Conditional result was False"} 2021-02-11T09:39:20.617 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2021-02-11T09:39:20.617 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2021-02-11T09:39:20.628 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2021-02-11T09:39:20.628 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item=loop3) => {"ansible_loop_var": "item", "changed": false, "item": "loop3", "skip_reason": "Conditional result was False"} 2021-02-11T09:39:20.639 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item=loop2) => {"ansible_loop_var": "item", "changed": false, "item": "loop2", "skip_reason": "Conditional result was False"} 2021-02-11T09:39:20.640 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2021-02-11T09:39:20.650 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item=loop0) => {"ansible_loop_var": "item", "changed": false, "item": "loop0", "skip_reason": "Conditional result was False"} 2021-02-11T09:39:20.661 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item=loop7) => {"ansible_loop_var": "item", "changed": false, "item": "loop7", "skip_reason": "Conditional result was False"} 2021-02-11T09:39:20.662 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2021-02-11T09:39:20.672 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item=loop5) => {"ansible_loop_var": "item", "changed": false, "item": "loop5", "skip_reason": "Conditional result was False"} 2021-02-11T09:39:20.683 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-02-11T09:39:20.722 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:20 +0000 (0:00:00.124) 0:01:46.642 ***** 2021-02-11T09:39:20.894 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-02-11T09:39:20.896 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "size": 578, "state": "file", "uid": 0} 2021-02-11T09:39:20.931 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:20 +0000 (0:00:00.209) 0:01:46.852 ***** 2021-02-11T09:39:20.959 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-02-11T09:39:20.960 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:20.994 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:20 +0000 (0:00:00.062) 0:01:46.915 ***** 2021-02-11T09:39:21.051 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2021-02-11T09:39:21.053 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.006917", "end": "2021-02-11 09:39:21.025133", "rc": 0, "start": "2021-02-11 09:39:21.018216", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/bin/teuthology-lock"]} 2021-02-11T09:39:21.092 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:21 +0000 (0:00:00.097) 0:01:47.012 ***** 2021-02-11T09:39:21.118 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2021-02-11T09:39:21.119 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:21.157 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:21 +0000 (0:00:00.065) 0:01:47.078 ***** 2021-02-11T09:39:21.184 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:21.221 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:21 +0000 (0:00:00.063) 0:01:47.141 ***** 2021-02-11T09:39:21.248 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2021-02-11T09:39:21.249 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:21.290 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:21 +0000 (0:00:00.069) 0:01:47.211 ***** 2021-02-11T09:39:21.323 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2021-02-11T09:39:21.324 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => {"changed": false, "checksum": "d5677bbd13857966eb0fc86f66195b18a2b98e20", "dest": "/etc/cron.weekly/kernel-clean", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cron.weekly/kernel-clean", "size": 752, "state": "file", "uid": 0} 2021-02-11T09:39:21.362 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:21 +0000 (0:00:00.071) 0:01:47.283 ***** 2021-02-11T09:39:21.703 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2021-02-11T09:39:21.705 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => {"changed": false, "checksum": "22e9a1446f43d0440219b9af14a66dd77fb0c2eb", "dest": "/etc/grub.d/02_force_timeout", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/grub.d/02_force_timeout", "size": 115, "state": "file", "uid": 0} 2021-02-11T09:39:21.738 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:21 +0000 (0:00:00.375) 0:01:47.658 ***** 2021-02-11T09:39:22.064 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2021-02-11T09:39:22.065 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => {"changed": false, "checksum": "6449466d762bced58744df80432d189e35329803", "dest": "/etc/modules", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/modules", "size": 315, "state": "file", "uid": 0} 2021-02-11T09:39:22.114 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:22 +0000 (0:00:00.376) 0:01:48.035 ***** 2021-02-11T09:39:22.183 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2021-02-11T09:39:22.184 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"cache_update_time": 1613036022, "cache_updated": false, "changed": false} 2021-02-11T09:39:22.219 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:22 +0000 (0:00:00.105) 0:01:48.140 ***** 2021-02-11T09:39:22.246 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2021-02-11T09:39:22.248 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:22.280 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:22 +0000 (0:00:00.060) 0:01:48.201 ***** 2021-02-11T09:39:22.286 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2021-02-11T09:39:22.287 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-11T09:39:22.322 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:22 +0000 (0:00:00.042) 0:01:48.243 ***** 2021-02-11T09:39:22.328 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-02-11T09:39:22.329 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'block_used': 13754874, 'uuid': '602a9f0b-ccf6-4d5d-877a-542e0952c07a', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226340234, 'size_available': 927089598464, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149545, 'block_size': 4096, 'inode_available': 60905431}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226340234, "block_size": 4096, "block_total": 240095108, "block_used": 13754874, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905431, "inode_total": 61054976, "inode_used": 149545, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927089598464, "size_total": 983429562368, "uuid": "602a9f0b-ccf6-4d5d-877a-542e0952c07a"}} 2021-02-11T09:39:22.365 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:22 +0000 (0:00:00.042) 0:01:48.285 ***** 2021-02-11T09:39:22.371 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2021-02-11T09:39:22.372 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:22.411 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:22 +0000 (0:00:00.046) 0:01:48.332 ***** 2021-02-11T09:39:22.438 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2021-02-11T09:39:22.439 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:22.473 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:22 +0000 (0:00:00.061) 0:01:48.394 ***** 2021-02-11T09:39:22.499 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2021-02-11T09:39:22.501 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.008132", "end": "2021-02-11 09:39:22.481889", "rc": 0, "start": "2021-02-11 09:39:22.473757", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:22.535 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:22 +0000 (0:00:00.061) 0:01:48.455 ***** 2021-02-11T09:39:22.570 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:22 +0000 (0:00:00.035) 0:01:48.491 ***** 2021-02-11T09:39:22.575 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2021-02-11T09:39:22.576 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'block_used': 13754874, 'uuid': '602a9f0b-ccf6-4d5d-877a-542e0952c07a', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226340234, 'size_available': 927089598464, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149545, 'block_size': 4096, 'inode_available': 60905431}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226340234, "block_size": 4096, "block_total": 240095108, "block_used": 13754874, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905431, "inode_total": 61054976, "inode_used": 149545, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927089598464, "size_total": 983429562368, "uuid": "602a9f0b-ccf6-4d5d-877a-542e0952c07a"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:22.611 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:22 +0000 (0:00:00.041) 0:01:48.532 ***** 2021-02-11T09:39:22.616 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2021-02-11T09:39:22.618 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:22.650 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:22 +0000 (0:00:00.038) 0:01:48.571 ***** 2021-02-11T09:39:22.657 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2021-02-11T09:39:22.697 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:22 +0000 (0:00:00.046) 0:01:48.618 ***** 2021-02-11T09:39:22.705 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2021-02-11T09:39:22.707 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:22.744 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:22 +0000 (0:00:00.046) 0:01:48.665 ***** 2021-02-11T09:39:22.751 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-02-11T09:39:22.751 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:22.773 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-11T09:39:22.774 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-02-11T09:39:22.808 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:22 +0000 (0:00:00.064) 0:01:48.729 ***** 2021-02-11T09:39:22.846 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2021-02-11T09:39:22.847 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item=loop1) => {"ansible_loop_var": "item", "changed": false, "item": "loop1", "skip_reason": "Conditional result was False"} 2021-02-11T09:39:22.857 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2021-02-11T09:39:22.857 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item=loop6) => {"ansible_loop_var": "item", "changed": false, "item": "loop6", "skip_reason": "Conditional result was False"} 2021-02-11T09:39:22.868 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item=loop4) => {"ansible_loop_var": "item", "changed": false, "item": "loop4", "skip_reason": "Conditional result was False"} 2021-02-11T09:39:22.878 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2021-02-11T09:39:22.879 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2021-02-11T09:39:22.890 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2021-02-11T09:39:22.900 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item=loop3) => {"ansible_loop_var": "item", "changed": false, "item": "loop3", "skip_reason": "Conditional result was False"} 2021-02-11T09:39:22.911 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item=loop2) => {"ansible_loop_var": "item", "changed": false, "item": "loop2", "skip_reason": "Conditional result was False"} 2021-02-11T09:39:22.922 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2021-02-11T09:39:22.922 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item=loop0) => {"ansible_loop_var": "item", "changed": false, "item": "loop0", "skip_reason": "Conditional result was False"} 2021-02-11T09:39:22.933 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item=loop7) => {"ansible_loop_var": "item", "changed": false, "item": "loop7", "skip_reason": "Conditional result was False"} 2021-02-11T09:39:22.933 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2021-02-11T09:39:22.944 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item=loop5) => {"ansible_loop_var": "item", "changed": false, "item": "loop5", "skip_reason": "Conditional result was False"} 2021-02-11T09:39:22.955 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-02-11T09:39:22.995 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:22 +0000 (0:00:00.186) 0:01:48.915 ***** 2021-02-11T09:39:23.176 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2021-02-11T09:39:23.178 INFO:teuthology.task.ansible.out:changed: [smithi098.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003198", "end": "2021-02-11 09:39:23.157202", "rc": 0, "start": "2021-02-11 09:39:23.154004", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/bin/teuthology-lock"]} 2021-02-11T09:39:23.214 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:23 +0000 (0:00:00.219) 0:01:49.135 ***** 2021-02-11T09:39:23.240 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2021-02-11T09:39:23.242 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:23.277 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:23 +0000 (0:00:00.062) 0:01:49.198 ***** 2021-02-11T09:39:23.303 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2021-02-11T09:39:23.305 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:23.336 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:23 +0000 (0:00:00.059) 0:01:49.257 ***** 2021-02-11T09:39:23.363 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2021-02-11T09:39:23.364 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:23.399 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:23 +0000 (0:00:00.062) 0:01:49.319 ***** 2021-02-11T09:39:24.285 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2021-02-11T09:39:24.286 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => {"cache_update_time": 1613036065, "cache_updated": false, "changed": false} 2021-02-11T09:39:24.318 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:24 +0000 (0:00:00.919) 0:01:50.239 ***** 2021-02-11T09:39:24.344 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2021-02-11T09:39:24.345 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:24.379 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:24 +0000 (0:00:00.061) 0:01:50.300 ***** 2021-02-11T09:39:24.416 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-02-11T09:39:24.417 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item={'block_used': 13754879, 'uuid': '602a9f0b-ccf6-4d5d-877a-542e0952c07a', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226340229, 'size_available': 927089577984, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149545, 'block_size': 4096, 'inode_available': 60905431}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226340229, "block_size": 4096, "block_total": 240095108, "block_used": 13754879, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905431, "inode_total": 61054976, "inode_used": 149545, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927089577984, "size_total": 983429562368, "uuid": "602a9f0b-ccf6-4d5d-877a-542e0952c07a"}} 2021-02-11T09:39:24.461 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:24 +0000 (0:00:00.082) 0:01:50.382 ***** 2021-02-11T09:39:24.591 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2021-02-11T09:39:24.593 INFO:teuthology.task.ansible.out:changed: [smithi098.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.009596", "end": "2021-02-11 09:39:24.834085", "rc": 0, "start": "2021-02-11 09:39:24.824489", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-11T09:39:24.626 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:24 +0000 (0:00:00.164) 0:01:50.547 ***** 2021-02-11T09:39:24.663 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2021-02-11T09:39:24.665 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'block_used': 13754879, 'uuid': '602a9f0b-ccf6-4d5d-877a-542e0952c07a', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226340229, 'size_available': 927089577984, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149545, 'block_size': 4096, 'inode_available': 60905431}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226340229, "block_size": 4096, "block_total": 240095108, "block_used": 13754879, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905431, "inode_total": 61054976, "inode_used": 149545, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927089577984, "size_total": 983429562368, "uuid": "602a9f0b-ccf6-4d5d-877a-542e0952c07a"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:24.698 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:24 +0000 (0:00:00.072) 0:01:50.619 ***** 2021-02-11T09:39:24.725 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2021-02-11T09:39:24.761 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:24 +0000 (0:00:00.062) 0:01:50.682 ***** 2021-02-11T09:39:24.811 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-02-11T09:39:24.812 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:24.824 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-LWcy3kiSAcGV2NAnffe4dvRkgHPCyxBgkKmnazjSGDNThRoye3OYFAQDaCdLrqQ2'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.953070", "end": "2021-02-11 09:39:24.794914", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-LWcy3kiSAcGV2NAnffe4dvRkgHPCyxBgkKmnazjSGDNThRoye3OYFAQDaCdLrqQ2"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 09:39:22.841844", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-11T09:39:24.835 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:24.836 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:26.091 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-LWcy3kiSAcGV2NAnffe4dvRkgHPCyxBgl1WH3QMFY1lee8WN2UCrodjEnxbJwnRE'], 'uuids': ['c3a92ec2-8548-429f-915c-8d343620bac8']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.125289", "end": "2021-02-11 09:39:26.063193", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-LWcy3kiSAcGV2NAnffe4dvRkgHPCyxBgl1WH3QMFY1lee8WN2UCrodjEnxbJwnRE"], "labels": [], "masters": [], "uuids": ["c3a92ec2-8548-429f-915c-8d343620bac8"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 09:39:24.937904", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-11T09:39:26.103 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5LJNP', 'wwn-0x5000c500918dbfac'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '602a9f0b-ccf6-4d5d-877a-542e0952c07a', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5LJNP-part1', 'wwn-0x5000c500918dbfac-part1'], 'uuids': ['602a9f0b-ccf6-4d5d-877a-542e0952c07a']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c500918dbfac', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5LJNP', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5LJNP", "wwn-0x5000c500918dbfac"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5LJNP-part1", "wwn-0x5000c500918dbfac-part1"], "labels": [], "masters": [], "uuids": ["602a9f0b-ccf6-4d5d-877a-542e0952c07a"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "602a9f0b-ccf6-4d5d-877a-542e0952c07a"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5LJNP", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500918dbfac"}}, "skip_reason": 2021-02-11T09:39:26.104 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2021-02-11T09:39:27.304 INFO:teuthology.task.ansible.out:changed: [smithi098.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-L6Al4qtLkiG4dzD6m2pR0U0E9LXezcZpcEcKl8MKKJZD8gJ9HrrwkFoZ33JDf92z'], 'uuids': ['5e9b251e-ac55-4dc0-8c60-d38de7fddcc2']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:02.349323", "end": "2021-02-11 09:39:27.531791", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-L6Al4qtLkiG4dzD6m2pR0U0E9LXezcZpcEcKl8MKKJZD8gJ9HrrwkFoZ33JDf92z"], "labels": [], "masters": [], "uuids": ["5e9b251e-ac55-4dc0-8c60-d38de7fddcc2"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 09:39:25.182468", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-11T09:39:27.315 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:27.316 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:27.369 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '937703088', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-fj4d7o-sHTS-Lo1B-1wRB-WuK8-u5f2-72EC9M', 'nvme-INTEL_SSDPED1D480GA_PHMB7513001L480DGN', 'nvme-nvme.8086-50484d42373531333030314c34383044474e-494e54454c2053534450454431443438304741-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPED1D480GA', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '447.13 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.152245", "end": "2021-02-11 09:39:27.343154", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series", "links": {"ids": ["lvm-pv-uuid-fj4d7o-sHTS-Lo1B-1wRB-WuK8-u5f2-72EC9M", "nvme-INTEL_SSDPED1D480GA_PHMB7513001L480DGN", "nvme-nvme.8086-50484d42373531333030314c34383044474e-494e54454c2053534450454431443438304741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPED1D480GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "937703088", "sectorsize": "512", "size": "447.13 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 09:39:26.190909", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! You may now pa 2021-02-11T09:39:27.371 INFO:teuthology.task.ansible.out:rtition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-11T09:39:27.381 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:27.392 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:28.606 INFO:teuthology.task.ansible.out:changed: [smithi098.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-L6Al4qtLkiG4dzD6m2pR0U0E9LXezcZpUcidpbfInxV8BJr07Inec3PL0NUcM0N1'], 'uuids': ['7eb23c91-54b4-434e-9ee7-383d77d2bfa2']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.177310", "end": "2021-02-11 09:39:28.840347", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-L6Al4qtLkiG4dzD6m2pR0U0E9LXezcZpUcidpbfInxV8BJr07Inec3PL0NUcM0N1"], "labels": [], "masters": [], "uuids": ["7eb23c91-54b4-434e-9ee7-383d77d2bfa2"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 09:39:27.663037", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-11T09:39:28.618 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5LEE6', 'wwn-0x5000c500918e39df'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '602a9f0b-ccf6-4d5d-877a-542e0952c07a', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5LEE6-part1', 'wwn-0x5000c500918e39df-part1'], 'uuids': ['602a9f0b-ccf6-4d5d-877a-542e0952c07a']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c500918e39df', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5LEE6', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5LEE6", "wwn-0x5000c500918e39df"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5LEE6-part1", "wwn-0x5000c500918e39df-part1"], "labels": [], "masters": [], "uuids": ["602a9f0b-ccf6-4d5d-877a-542e0952c07a"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "602a9f0b-ccf6-4d5d-877a-542e0952c07a"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5LEE6", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500918e39df"}}, "skip_reason": 2021-02-11T09:39:28.619 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2021-02-11T09:39:28.661 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-LWcy3kiSAcGV2NAnffe4dvRkgHPCyxBgB1TJLQT089hCtEdyLwhUa4bzxcf1HFAF'], 'uuids': ['266a5f4f-3435-4490-974a-e6f5afd3bc5f']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.156849", "end": "2021-02-11 09:39:28.631704", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-LWcy3kiSAcGV2NAnffe4dvRkgHPCyxBgB1TJLQT089hCtEdyLwhUa4bzxcf1HFAF"], "labels": [], "masters": [], "uuids": ["266a5f4f-3435-4490-974a-e6f5afd3bc5f"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 09:39:27.474855", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-11T09:39:28.672 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:28.673 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:29.904 INFO:teuthology.task.ansible.out:changed: [smithi098.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-k0AFs7-Xxpv-aSu3-ZAe1-DROD-utCI-xvpMVE', 'nvme-INTEL_SSDPEDMD400G4_PHFT6204010H400BGN', 'nvme-nvme.8086-50484654363230343031304834303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.169363", "end": "2021-02-11 09:39:30.131375", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-k0AFs7-Xxpv-aSu3-ZAe1-DROD-utCI-xvpMVE", "nvme-INTEL_SSDPEDMD400G4_PHFT6204010H400BGN", "nvme-nvme.8086-50484654363230343031304834303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 09:39:28.962012", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2021-02-11T09:39:29.906 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} skipping: [smithi098.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:29.917 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:29.959 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-LWcy3kiSAcGV2NAnffe4dvRkgHPCyxBg2eczlZkGOXPxcJlac8BM2HdoocpGl0fk'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.156214", "end": "2021-02-11 09:39:29.931807", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-LWcy3kiSAcGV2NAnffe4dvRkgHPCyxBg2eczlZkGOXPxcJlac8BM2HdoocpGl0fk"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 09:39:28.775593", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-11T09:39:29.971 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:31.231 INFO:teuthology.task.ansible.out:changed: [smithi098.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-L6Al4qtLkiG4dzD6m2pR0U0E9LXezcZp5cEstRDyR4rITwomRpmvBWF2eH9wQsfY'], 'uuids': ['7710653d-af6c-460c-a67d-94bc18472f4f']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.195068", "end": "2021-02-11 09:39:31.454870", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-L6Al4qtLkiG4dzD6m2pR0U0E9LXezcZp5cEstRDyR4rITwomRpmvBWF2eH9wQsfY"], "labels": [], "masters": [], "uuids": ["7710653d-af6c-460c-a67d-94bc18472f4f"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 09:39:30.259802", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-11T09:39:31.232 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:31.243 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-LWcy3kiSAcGV2NAnffe4dvRkgHPCyxBg0KZVyXrGhiz023Qc7uAk7OuK35sjb0fU'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.152427", "end": "2021-02-11 09:39:31.216347", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-LWcy3kiSAcGV2NAnffe4dvRkgHPCyxBg0KZVyXrGhiz023Qc7uAk7OuK35sjb0fU"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 09:39:30.063920", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-11T09:39:31.254 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:31.291 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:31 +0000 (0:00:06.529) 0:01:57.211 ***** 2021-02-11T09:39:31.329 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-02-11T09:39:31.330 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:31.579 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-LWcy3kiSAcGV2NAnffe4dvRkgHPCyxBgkKmnazjSGDNThRoye3OYFAQDaCdLrqQ2'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.119714", "end": "2021-02-11 09:39:31.554418", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-LWcy3kiSAcGV2NAnffe4dvRkgHPCyxBgkKmnazjSGDNThRoye3OYFAQDaCdLrqQ2"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 09:39:31.434704", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.114895 s, 1.0 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.114895 s, 1.0 GB/s"], "stdout": "", "stdout_lines": []} 2021-02-11T09:39:31.591 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:31.592 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:31.831 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-LWcy3kiSAcGV2NAnffe4dvRkgHPCyxBgl1WH3QMFY1lee8WN2UCrodjEnxbJwnRE'], 'uuids': ['c3a92ec2-8548-429f-915c-8d343620bac8']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.113323", "end": "2021-02-11 09:39:31.804116", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-LWcy3kiSAcGV2NAnffe4dvRkgHPCyxBgl1WH3QMFY1lee8WN2UCrodjEnxbJwnRE"], "labels": [], "masters": [], "uuids": ["c3a92ec2-8548-429f-915c-8d343620bac8"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 09:39:31.690793", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.108735 s, 1.1 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.108735 s, 1.1 GB/s"], "stdout": "", "stdout_lines": []} 2021-02-11T09:39:31.833 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5LJNP', 'wwn-0x5000c500918dbfac'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '602a9f0b-ccf6-4d5d-877a-542e0952c07a', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5LJNP-part1', 'wwn-0x5000c500918dbfac-part1'], 'uuids': ['602a9f0b-ccf6-4d5d-877a-542e0952c07a']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c500918dbfac', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5LJNP', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5LJNP", "wwn-0x5000c500918dbfac"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5LJNP-part1", "wwn-0x5000c500918dbfac-part1"], "labels": [], "masters": [], "uuids": ["602a9f0b-ccf6-4d5d-877a-542e0952c07a"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "602a9f0b-ccf6-4d5d-877a-542e0952c07a"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5LJNP", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500918dbfac"}}, "skip_reason": 2021-02-11T09:39:31.834 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2021-02-11T09:39:32.000 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '937703088', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-fj4d7o-sHTS-Lo1B-1wRB-WuK8-u5f2-72EC9M', 'nvme-INTEL_SSDPED1D480GA_PHMB7513001L480DGN', 'nvme-nvme.8086-50484d42373531333030314c34383044474e-494e54454c2053534450454431443438304741-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPED1D480GA', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '447.13 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.049669", "end": "2021-02-11 09:39:31.980729", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series", "links": {"ids": ["lvm-pv-uuid-fj4d7o-sHTS-Lo1B-1wRB-WuK8-u5f2-72EC9M", "nvme-INTEL_SSDPED1D480GA_PHMB7513001L480DGN", "nvme-nvme.8086-50484d42373531333030314c34383044474e-494e54454c2053534450454431443438304741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPED1D480GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "937703088", "sectorsize": "512", "size": "447.13 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 09:39:31.931060", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0468271 s, 2.5 GB/s", "stderr_lines": [" 2021-02-11T09:39:32.001 INFO:teuthology.task.ansible.out:110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0468271 s, 2.5 GB/s"], "stdout": "", "stdout_lines": []} 2021-02-11T09:39:32.012 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:32.023 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:32.242 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-LWcy3kiSAcGV2NAnffe4dvRkgHPCyxBgB1TJLQT089hCtEdyLwhUa4bzxcf1HFAF'], 'uuids': ['266a5f4f-3435-4490-974a-e6f5afd3bc5f']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.105311", "end": "2021-02-11 09:39:32.216906", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-LWcy3kiSAcGV2NAnffe4dvRkgHPCyxBgB1TJLQT089hCtEdyLwhUa4bzxcf1HFAF"], "labels": [], "masters": [], "uuids": ["266a5f4f-3435-4490-974a-e6f5afd3bc5f"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 09:39:32.111595", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.102146 s, 1.1 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.102146 s, 1.1 GB/s"], "stdout": "", "stdout_lines": []} 2021-02-11T09:39:32.254 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:32.255 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:32.494 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-LWcy3kiSAcGV2NAnffe4dvRkgHPCyxBg2eczlZkGOXPxcJlac8BM2HdoocpGl0fk'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.112034", "end": "2021-02-11 09:39:32.465004", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-LWcy3kiSAcGV2NAnffe4dvRkgHPCyxBg2eczlZkGOXPxcJlac8BM2HdoocpGl0fk"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 09:39:32.352970", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.107397 s, 1.1 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.107397 s, 1.1 GB/s"], "stdout": "", "stdout_lines": []} 2021-02-11T09:39:32.495 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-02-11T09:39:32.496 INFO:teuthology.task.ansible.out:changed: [smithi098.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-L6Al4qtLkiG4dzD6m2pR0U0E9LXezcZpMLfyTk2cHsQLOhzWG8flV8Pm36GOAvSN'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.137082", "end": "2021-02-11 09:39:32.721055", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-L6Al4qtLkiG4dzD6m2pR0U0E9LXezcZpMLfyTk2cHsQLOhzWG8flV8Pm36GOAvSN"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 09:39:31.583973", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-11T09:39:32.497 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** skipping: [smithi094.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:32.498 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** skipping: [smithi098.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:32.728 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-02-11T09:39:32.728 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-LWcy3kiSAcGV2NAnffe4dvRkgHPCyxBg0KZVyXrGhiz023Qc7uAk7OuK35sjb0fU'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.111071", "end": "2021-02-11 09:39:32.703033", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-LWcy3kiSAcGV2NAnffe4dvRkgHPCyxBg0KZVyXrGhiz023Qc7uAk7OuK35sjb0fU"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 09:39:32.591962", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.106316 s, 1.1 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.106316 s, 1.1 GB/s"], "stdout": "", "stdout_lines": []} 2021-02-11T09:39:32.777 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:32 +0000 (0:00:01.486) 0:01:58.698 ***** 2021-02-11T09:39:33.053 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2021-02-11T09:39:33.054 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.146576", "end": "2021-02-11 09:39:33.031193", "rc": 0, "start": "2021-02-11 09:39:32.884617", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-11T09:39:33.085 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:33 +0000 (0:00:00.308) 0:01:59.006 ***** 2021-02-11T09:39:33.267 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2021-02-11T09:39:33.268 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.049911", "end": "2021-02-11 09:39:33.243933", "rc": 0, "start": "2021-02-11 09:39:33.194022", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-11T09:39:33.302 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:33 +0000 (0:00:00.217) 0:01:59.223 ***** 2021-02-11T09:39:33.330 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-02-11T09:39:33.369 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:33 +0000 (0:00:00.066) 0:01:59.289 ***** 2021-02-11T09:39:33.406 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:33.448 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:33 +0000 (0:00:00.079) 0:01:59.369 ***** 2021-02-11T09:39:33.478 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2021-02-11T09:39:33.479 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:33.515 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:33 +0000 (0:00:00.067) 0:01:59.436 ***** 2021-02-11T09:39:33.542 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:33.580 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:33 +0000 (0:00:00.064) 0:01:59.500 ***** 2021-02-11T09:39:33.606 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2021-02-11T09:39:33.608 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:33.648 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:33 +0000 (0:00:00.068) 0:01:59.569 ***** 2021-02-11T09:39:33.759 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-02-11T09:39:33.760 INFO:teuthology.task.ansible.out:changed: [smithi098.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-L6Al4qtLkiG4dzD6m2pR0U0E9LXezcZpzsMuKMOm2V2HZai8RR6Y5i3s2TWI0Y8V'], 'uuids': ['96b51b71-2ab3-4935-8ae5-2c2731686207']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.134236", "end": "2021-02-11 09:39:33.988148", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-L6Al4qtLkiG4dzD6m2pR0U0E9LXezcZpzsMuKMOm2V2HZai8RR6Y5i3s2TWI0Y8V"], "labels": [], "masters": [], "uuids": ["96b51b71-2ab3-4935-8ae5-2c2731686207"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 09:39:32.853912", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-11T09:39:33.781 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2021-02-11T09:39:33.782 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-11T09:39:33.822 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:33 +0000 (0:00:00.174) 0:01:59.743 ***** 2021-02-11T09:39:33.856 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:33 +0000 (0:00:00.033) 0:01:59.776 ***** 2021-02-11T09:39:33.873 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-02-11T09:39:33.873 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'block_used': 13754874, 'uuid': '602a9f0b-ccf6-4d5d-877a-542e0952c07a', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226340234, 'size_available': 927089598464, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149545, 'block_size': 4096, 'inode_available': 60905431}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226340234, "block_size": 4096, "block_total": 240095108, "block_used": 13754874, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905431, "inode_total": 61054976, "inode_used": 149545, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927089598464, "size_total": 983429562368, "uuid": "602a9f0b-ccf6-4d5d-877a-542e0952c07a"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:33.913 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:33 +0000 (0:00:00.057) 0:01:59.833 ***** 2021-02-11T09:39:33.919 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-02-11T09:39:33.920 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:33.942 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2021-02-11T09:39:33.942 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:33.976 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:33 +0000 (0:00:00.063) 0:01:59.897 ***** 2021-02-11T09:39:34.003 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:34.036 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:34 +0000 (0:00:00.059) 0:01:59.956 ***** 2021-02-11T09:39:34.062 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-11T09:39:34.063 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:34.093 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:34 +0000 (0:00:00.057) 0:02:00.014 ***** 2021-02-11T09:39:34.119 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:34.155 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:34 +0000 (0:00:00.061) 0:02:00.075 ***** 2021-02-11T09:39:34.174 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-02-11T09:39:34.174 INFO:teuthology.task.ansible.out:changed: [smithi098.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-L6Al4qtLkiG4dzD6m2pR0U0E9LXezcZpcEcKl8MKKJZD8gJ9HrrwkFoZ33JDf92z'], 'uuids': ['5e9b251e-ac55-4dc0-8c60-d38de7fddcc2']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.147253", "end": "2021-02-11 09:39:34.391781", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-L6Al4qtLkiG4dzD6m2pR0U0E9LXezcZpcEcKl8MKKJZD8gJ9HrrwkFoZ33JDf92z"], "labels": [], "masters": [], "uuids": ["5e9b251e-ac55-4dc0-8c60-d38de7fddcc2"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 09:39:34.244528", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.144468 s, 798 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.144468 s, 798 MB/s"], "stdout": "", "stdout_lines": []} 2021-02-11T09:39:34.176 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:34.187 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2021-02-11T09:39:34.188 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** skipping: [smithi098.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:34.222 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:34 +0000 (0:00:00.067) 0:02:00.142 ***** 2021-02-11T09:39:34.251 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:34.284 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:34 +0000 (0:00:00.062) 0:02:00.205 ***** 2021-02-11T09:39:34.469 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-02-11T09:39:34.470 INFO:teuthology.task.ansible.out:changed: [smithi098.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-L6Al4qtLkiG4dzD6m2pR0U0E9LXezcZpUcidpbfInxV8BJr07Inec3PL0NUcM0N1'], 'uuids': ['7eb23c91-54b4-434e-9ee7-383d77d2bfa2']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.164107", "end": "2021-02-11 09:39:34.697270", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-L6Al4qtLkiG4dzD6m2pR0U0E9LXezcZpUcidpbfInxV8BJr07Inec3PL0NUcM0N1"], "labels": [], "masters": [], "uuids": ["7eb23c91-54b4-434e-9ee7-383d77d2bfa2"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 09:39:34.533163", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.159896 s, 721 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.159896 s, 721 MB/s"], "stdout": "", "stdout_lines": []} 2021-02-11T09:39:34.481 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5LEE6', 'wwn-0x5000c500918e39df'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '602a9f0b-ccf6-4d5d-877a-542e0952c07a', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5LEE6-part1', 'wwn-0x5000c500918e39df-part1'], 'uuids': ['602a9f0b-ccf6-4d5d-877a-542e0952c07a']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c500918e39df', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5LEE6', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5LEE6", "wwn-0x5000c500918e39df"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5LEE6-part1", "wwn-0x5000c500918e39df-part1"], "labels": [], "masters": [], "uuids": ["602a9f0b-ccf6-4d5d-877a-542e0952c07a"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "602a9f0b-ccf6-4d5d-877a-542e0952c07a"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5LEE6", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500918e39df"}}, "skip_reason": 2021-02-11T09:39:34.482 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2021-02-11T09:39:34.648 INFO:teuthology.task.ansible.out:changed: [smithi098.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-k0AFs7-Xxpv-aSu3-ZAe1-DROD-utCI-xvpMVE', 'nvme-INTEL_SSDPEDMD400G4_PHFT6204010H400BGN', 'nvme-nvme.8086-50484654363230343031304834303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.047209", "end": "2021-02-11 09:39:34.877099", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-k0AFs7-Xxpv-aSu3-ZAe1-DROD-utCI-xvpMVE", "nvme-INTEL_SSDPEDMD400G4_PHFT6204010H400BGN", "nvme-nvme.8086-50484654363230343031304834303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 09:39:34.829890", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0425161 s, 2.7 GB/s", "std 2021-02-11T09:39:34.649 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0425161 s, 2.7 GB/s"], "stdout": "", "stdout_lines": []} 2021-02-11T09:39:34.659 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:34.671 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:34.878 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2021-02-11T09:39:34.879 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-02-11T09:39:34.914 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:34 +0000 (0:00:00.629) 0:02:00.835 ***** 2021-02-11T09:39:34.923 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-02-11T09:39:34.924 INFO:teuthology.task.ansible.out:changed: [smithi098.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-L6Al4qtLkiG4dzD6m2pR0U0E9LXezcZp5cEstRDyR4rITwomRpmvBWF2eH9wQsfY'], 'uuids': ['7710653d-af6c-460c-a67d-94bc18472f4f']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.149266", "end": "2021-02-11 09:39:35.152452", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-L6Al4qtLkiG4dzD6m2pR0U0E9LXezcZp5cEstRDyR4rITwomRpmvBWF2eH9wQsfY"], "labels": [], "masters": [], "uuids": ["7710653d-af6c-460c-a67d-94bc18472f4f"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 09:39:35.003186", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.146296 s, 788 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.146296 s, 788 MB/s"], "stdout": "", "stdout_lines": []} 2021-02-11T09:39:34.937 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:34.938 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:35.201 INFO:teuthology.task.ansible.out:changed: [smithi098.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-L6Al4qtLkiG4dzD6m2pR0U0E9LXezcZpMLfyTk2cHsQLOhzWG8flV8Pm36GOAvSN'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.149452", "end": "2021-02-11 09:39:35.428375", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-L6Al4qtLkiG4dzD6m2pR0U0E9LXezcZpMLfyTk2cHsQLOhzWG8flV8Pm36GOAvSN"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 09:39:35.278923", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.146974 s, 785 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.146974 s, 785 MB/s"], "stdout": "", "stdout_lines": []} 2021-02-11T09:39:35.212 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:35.473 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* changed: [smithi094.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-11T09:39:35.484 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-02-11T09:39:35.484 INFO:teuthology.task.ansible.out:changed: [smithi098.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-L6Al4qtLkiG4dzD6m2pR0U0E9LXezcZpzsMuKMOm2V2HZai8RR6Y5i3s2TWI0Y8V'], 'uuids': ['96b51b71-2ab3-4935-8ae5-2c2731686207']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.150926", "end": "2021-02-11 09:39:35.712715", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-L6Al4qtLkiG4dzD6m2pR0U0E9LXezcZpzsMuKMOm2V2HZai8RR6Y5i3s2TWI0Y8V"], "labels": [], "masters": [], "uuids": ["96b51b71-2ab3-4935-8ae5-2c2731686207"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 09:39:35.561789", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.147822 s, 780 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.147822 s, 780 MB/s"], "stdout": "", "stdout_lines": []} 2021-02-11T09:39:35.532 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:35 +0000 (0:00:00.617) 0:02:01.452 ***** 2021-02-11T09:39:35.694 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-02-11T09:39:35.696 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-11T09:39:35.799 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2021-02-11T09:39:35.801 INFO:teuthology.task.ansible.out:changed: [smithi098.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.137782", "end": "2021-02-11 09:39:36.027558", "rc": 0, "start": "2021-02-11 09:39:35.889776", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-11T09:39:35.836 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:35 +0000 (0:00:00.303) 0:02:01.756 ***** 2021-02-11T09:39:35.914 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-02-11T09:39:35.916 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-11T09:39:36.009 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2021-02-11T09:39:36.010 INFO:teuthology.task.ansible.out:changed: [smithi098.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.052094", "end": "2021-02-11 09:39:36.242749", "rc": 0, "start": "2021-02-11 09:39:36.190655", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-11T09:39:36.044 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:36 +0000 (0:00:00.208) 0:02:01.965 ***** 2021-02-11T09:39:36.070 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-02-11T09:39:36.107 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:36 +0000 (0:00:00.063) 0:02:02.028 ***** 2021-02-11T09:39:36.134 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2021-02-11T09:39:36.136 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Create logical volume(s)] ************************************* changed: [smithi094.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-11T09:39:36.171 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:36 +0000 (0:00:00.063) 0:02:02.092 ***** 2021-02-11T09:39:36.197 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2021-02-11T09:39:36.199 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:36.232 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:36 +0000 (0:00:00.061) 0:02:02.153 ***** 2021-02-11T09:39:36.258 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-02-11T09:39:36.260 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:36.295 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:36 +0000 (0:00:00.063) 0:02:02.216 ***** 2021-02-11T09:39:36.322 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2021-02-11T09:39:36.324 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:36.357 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:36 +0000 (0:00:00.061) 0:02:02.278 ***** 2021-02-11T09:39:36.373 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-02-11T09:39:36.375 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-11T09:39:36.414 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:36 +0000 (0:00:00.056) 0:02:02.334 ***** 2021-02-11T09:39:36.472 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2021-02-11T09:39:36.473 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-11T09:39:36.508 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:36 +0000 (0:00:00.094) 0:02:02.429 ***** 2021-02-11T09:39:36.535 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-02-11T09:39:36.537 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-02-11T09:39:36.572 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:36 +0000 (0:00:00.064) 0:02:02.493 ***** 2021-02-11T09:39:36.579 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-02-11T09:39:36.580 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'block_used': 13754879, 'uuid': '602a9f0b-ccf6-4d5d-877a-542e0952c07a', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226340229, 'size_available': 927089577984, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149545, 'block_size': 4096, 'inode_available': 60905431}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226340229, "block_size": 4096, "block_total": 240095108, "block_used": 13754879, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905431, "inode_total": 61054976, "inode_used": 149545, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927089577984, "size_total": 983429562368, "uuid": "602a9f0b-ccf6-4d5d-877a-542e0952c07a"}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:36.622 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:36 +0000 (0:00:00.049) 0:02:02.542 ***** 2021-02-11T09:39:36.648 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2021-02-11T09:39:36.649 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:36.682 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:36 +0000 (0:00:00.060) 0:02:02.603 ***** 2021-02-11T09:39:36.709 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-11T09:39:36.710 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:36.746 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:36 +0000 (0:00:00.063) 0:02:02.667 ***** 2021-02-11T09:39:36.754 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** changed: [smithi094.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-11T09:39:36.775 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-11T09:39:36.776 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:36.810 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:36 +0000 (0:00:00.063) 0:02:02.730 ***** 2021-02-11T09:39:36.826 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-02-11T09:39:36.826 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-11T09:39:36.838 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:36.873 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:36 +0000 (0:00:00.063) 0:02:02.794 ***** 2021-02-11T09:39:36.900 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2021-02-11T09:39:36.901 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:36.934 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:36 +0000 (0:00:00.060) 0:02:02.855 ***** 2021-02-11T09:39:36.940 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-02-11T09:39:36.940 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-11T09:39:36.963 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2021-02-11T09:39:36.963 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:37.008 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:37 +0000 (0:00:00.073) 0:02:02.928 ***** 2021-02-11T09:39:37.047 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-02-11T09:39:37.047 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-11T09:39:37.059 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:37.098 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:37 +0000 (0:00:00.090) 0:02:03.019 ***** 2021-02-11T09:39:37.125 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* 2021-02-11T09:39:37.127 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:37.163 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:37 +0000 (0:00:00.064) 0:02:03.083 ***** 2021-02-11T09:39:37.189 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** 2021-02-11T09:39:37.191 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:37.222 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:37 +0000 (0:00:00.059) 0:02:03.143 ***** 2021-02-11T09:39:37.343 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2021-02-11T09:39:37.343 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "size": 4096, "state": "directory", "uid": 0} 2021-02-11T09:39:37.379 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:37 +0000 (0:00:00.156) 0:02:03.300 ***** 2021-02-11T09:39:37.388 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2021-02-11T09:39:37.388 INFO:teuthology.task.ansible.out:changed: [smithi098.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-02-11T09:39:37.429 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:37 +0000 (0:00:00.050) 0:02:03.350 ***** 2021-02-11T09:39:37.438 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2021-02-11T09:39:37.438 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2021-02-11T09:39:37.482 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:37 +0000 (0:00:00.052) 0:02:03.402 ***** 2021-02-11T09:39:37.748 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-02-11T09:39:37.749 INFO:teuthology.task.ansible.out:changed: [smithi098.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-11T09:39:37.894 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2021-02-11T09:39:37.896 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => {"changed": true} 2021-02-11T09:39:37.931 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:37 +0000 (0:00:00.449) 0:02:03.852 ***** 2021-02-11T09:39:37.969 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-02-11T09:39:37.970 INFO:teuthology.task.ansible.out:changed: [smithi098.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-11T09:39:38.180 INFO:teuthology.task.ansible.out:changed: [smithi098.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-11T09:39:38.378 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2021-02-11T09:39:38.379 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-02-11T09:39:38.413 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:38 +0000 (0:00:00.481) 0:02:04.334 ***** 2021-02-11T09:39:38.418 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-02-11T09:39:38.419 INFO:teuthology.task.ansible.out:changed: [smithi098.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-11T09:39:38.440 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd] ****************************************** skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:38.475 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:38 +0000 (0:00:00.061) 0:02:04.396 ***** 2021-02-11T09:39:38.501 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd conf file] ******************************** 2021-02-11T09:39:38.503 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:38.536 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:38 +0000 (0:00:00.061) 0:02:04.457 ***** 2021-02-11T09:39:38.563 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:38.597 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:38 +0000 (0:00:00.060) 0:02:04.518 ***** 2021-02-11T09:39:38.623 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-02-11T09:39:38.625 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:38.659 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:38 +0000 (0:00:00.061) 0:02:04.579 ***** 2021-02-11T09:39:38.665 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-02-11T09:39:38.665 INFO:teuthology.task.ansible.out:changed: [smithi098.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-11T09:39:38.704 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:38 +0000 (0:00:00.045) 0:02:04.625 ***** 2021-02-11T09:39:38.804 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2021-02-11T09:39:38.806 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 1000} 2021-02-11T09:39:38.841 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:38 +0000 (0:00:00.136) 0:02:04.761 ***** 2021-02-11T09:39:38.847 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-02-11T09:39:38.848 INFO:teuthology.task.ansible.out:changed: [smithi098.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-02-11T09:39:38.884 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:38 +0000 (0:00:00.043) 0:02:04.805 ***** 2021-02-11T09:39:38.972 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2021-02-11T09:39:38.973 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 0} 2021-02-11T09:39:39.008 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:39 +0000 (0:00:00.123) 0:02:04.928 ***** 2021-02-11T09:39:39.025 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-02-11T09:39:39.026 INFO:teuthology.task.ansible.out:changed: [smithi098.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-11T09:39:39.151 INFO:teuthology.task.ansible.out:changed: [smithi098.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-11T09:39:39.258 INFO:teuthology.task.ansible.out:changed: [smithi098.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-11T09:39:39.321 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2021-02-11T09:39:39.323 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "gid": 1238, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 1000} 2021-02-11T09:39:39.357 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:39 +0000 (0:00:00.348) 0:02:05.277 ***** 2021-02-11T09:39:39.373 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-02-11T09:39:39.375 INFO:teuthology.task.ansible.out:changed: [smithi098.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-11T09:39:39.385 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.424 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:39 +0000 (0:00:00.067) 0:02:05.345 ***** 2021-02-11T09:39:39.452 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* 2021-02-11T09:39:39.455 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.488 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:39 +0000 (0:00:00.064) 0:02:05.409 ***** 2021-02-11T09:39:39.516 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** 2021-02-11T09:39:39.517 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:39.551 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:39 +0000 (0:00:00.062) 0:02:05.471 ***** 2021-02-11T09:39:39.672 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2021-02-11T09:39:39.673 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/root/.cpan/CPAN/MyConfig.pm", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 0} 2021-02-11T09:39:39.707 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:39 +0000 (0:00:00.156) 0:02:05.628 ***** 2021-02-11T09:39:39.713 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2021-02-11T09:39:39.713 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "size": 4096, "state": "directory", "uid": 0} 2021-02-11T09:39:39.754 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:39 +0000 (0:00:00.046) 0:02:05.674 ***** 2021-02-11T09:39:39.791 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2021-02-11T09:39:39.792 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2021-02-11T09:39:39.827 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:39 +0000 (0:00:00.073) 0:02:05.747 ***** 2021-02-11T09:39:40.607 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2021-02-11T09:39:40.608 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"cache_update_time": 1613036022, "cache_updated": false, "changed": false} 2021-02-11T09:39:40.648 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:40 +0000 (0:00:00.820) 0:02:06.568 ***** 2021-02-11T09:39:40.820 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2021-02-11T09:39:40.821 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "cmd": ["perldoc", "-l", "Amazon::S3"], "delta": "0:00:00.046164", "end": "2021-02-11 09:39:40.802577", "rc": 0, "start": "2021-02-11 09:39:40.756413", "stderr": "", "stderr_lines": [], "stdout": "/usr/local/share/perl/5.26.1/Amazon/S3.pm", "stdout_lines": ["/usr/local/share/perl/5.26.1/Amazon/S3.pm"]} 2021-02-11T09:39:40.855 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:40 +0000 (0:00:00.207) 0:02:06.776 ***** 2021-02-11T09:39:40.882 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2021-02-11T09:39:40.883 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.918 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:40 +0000 (0:00:00.063) 0:02:06.839 ***** 2021-02-11T09:39:40.946 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2021-02-11T09:39:40.948 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:40.983 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:40 +0000 (0:00:00.064) 0:02:06.904 ***** 2021-02-11T09:39:41.051 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2021-02-11T09:39:41.053 INFO:teuthology.task.ansible.out:changed: [smithi098.front.sepia.ceph.com] => {"changed": true} 2021-02-11T09:39:41.089 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:41 +0000 (0:00:00.105) 0:02:07.009 ***** 2021-02-11T09:39:41.303 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2021-02-11T09:39:41.305 INFO:teuthology.task.ansible.out:changed: [smithi098.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-02-11T09:39:41.340 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:41 +0000 (0:00:00.251) 0:02:07.261 ***** 2021-02-11T09:39:41.366 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd] ****************************************** 2021-02-11T09:39:41.368 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:41.400 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:41 +0000 (0:00:00.060) 0:02:07.321 ***** 2021-02-11T09:39:41.426 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd conf file] ******************************** 2021-02-11T09:39:41.427 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:41.463 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:41 +0000 (0:00:00.062) 0:02:07.383 ***** 2021-02-11T09:39:41.488 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-02-11T09:39:41.490 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:41.526 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:41 +0000 (0:00:00.063) 0:02:07.447 ***** 2021-02-11T09:39:41.553 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-02-11T09:39:41.554 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:41.588 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:41 +0000 (0:00:00.061) 0:02:07.508 ***** 2021-02-11T09:39:41.728 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2021-02-11T09:39:41.729 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 1000} 2021-02-11T09:39:41.762 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:41 +0000 (0:00:00.174) 0:02:07.682 ***** 2021-02-11T09:39:41.860 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2021-02-11T09:39:41.862 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"cache_update_time": 1613036022, "cache_updated": false, "changed": false} 2021-02-11T09:39:41.898 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:41 +0000 (0:00:00.135) 0:02:07.818 ***** 2021-02-11T09:39:41.914 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2021-02-11T09:39:41.916 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 0} 2021-02-11T09:39:41.950 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:41 +0000 (0:00:00.052) 0:02:07.871 ***** 2021-02-11T09:39:41.956 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2021-02-11T09:39:41.958 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:41.994 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:41 +0000 (0:00:00.043) 0:02:07.914 ***** 2021-02-11T09:39:42.270 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2021-02-11T09:39:42.272 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "gid": 1238, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 1000} 2021-02-11T09:39:42.306 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:42 +0000 (0:00:00.312) 0:02:08.227 ***** 2021-02-11T09:39:42.312 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2021-02-11T09:39:42.313 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "size": 2767, "state": "file", "uid": 0} 2021-02-11T09:39:42.350 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:42 +0000 (0:00:00.043) 0:02:08.270 ***** 2021-02-11T09:39:42.376 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:42.410 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:42 +0000 (0:00:00.060) 0:02:08.331 ***** 2021-02-11T09:39:42.603 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2021-02-11T09:39:42.604 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "ntp", "state": "started", "status": {"ActiveEnterTimestamp": "Thu 2021-02-11 09:37:04 UTC", "ActiveEnterTimestampMonotonic": "28724075", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-journald.socket basic.target system.slice -.mount network.target sysinit.target systemd-tmpfiles-setup.service", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2021-02-11 09:36:58 UTC", "AssertTimestampMonotonic": "21992505", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2021-02-11 09:36:58 UTC", "ConditionTimestampMonotonic": "21992504", "ConfigurationDirectoryMode": "0755", "Conflicts": "systemd-timesyncd.service shutdown.target", "ControlGroup": "/system.slice/ntp.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "Documen 2021-02-11T09:39:42.606 INFO:teuthology.task.ansible.out:tation": "man:ntpd(8)", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1646", "ExecMainStartTimestamp": "Thu 2021-02-11 09:37:04 UTC", "ExecMainStartTimestampMonotonic": "28724042", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/ntp.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "ntp.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2021-02-11 09:36:58 UTC", "InactiveExitTimestampMonotonic": "21994108", "InvocationID": "019b96c6cef7435180f258c62f7a87fb", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128029", "LimitNPROCSoft": "128029", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "Limi 2021-02-11T09:39:42.607 INFO:teuthology.task.ansible.out:tRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "128029", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1646", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "ntp.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount system.slice sysinit.target", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2021-02-11 09:37:04 UTC", "S 2021-02-11T09:39:42.609 INFO:teuthology.task.ansible.out:tateChangeTimestampMonotonic": "28724075", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "2", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2021-02-11 09:37:04 UTC", "WatchdogTimestampMonotonic": "28724073", "WatchdogUSec": "0"}} 2021-02-11T09:39:42.641 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:42 +0000 (0:00:00.230) 0:02:08.561 ***** 2021-02-11T09:39:42.647 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2021-02-11T09:39:42.647 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/root/.cpan/CPAN/MyConfig.pm", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 0} 2021-02-11T09:39:42.684 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:42 +0000 (0:00:00.042) 0:02:08.604 ***** 2021-02-11T09:39:42.689 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-11T09:39:42.689 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-02-11T09:39:42.726 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:42 +0000 (0:00:00.042) 0:02:08.647 ***** 2021-02-11T09:39:42.753 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-11T09:39:42.754 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:42.789 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:42 +0000 (0:00:00.062) 0:02:08.709 ***** 2021-02-11T09:39:42.815 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2021-02-11T09:39:42.816 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:42.849 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:42 +0000 (0:00:00.060) 0:02:08.770 ***** 2021-02-11T09:39:43.626 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2021-02-11T09:39:43.627 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => {"cache_update_time": 1613036065, "cache_updated": false, "changed": false} 2021-02-11T09:39:43.679 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:43 +0000 (0:00:00.829) 0:02:09.599 ***** 2021-02-11T09:39:43.727 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2021-02-11T09:39:43.729 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"cache_update_time": 1613036022, "cache_updated": false, "changed": false} 2021-02-11T09:39:43.766 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:43 +0000 (0:00:00.087) 0:02:09.687 ***** 2021-02-11T09:39:43.793 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2021-02-11T09:39:43.794 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:43.828 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:43 +0000 (0:00:00.062) 0:02:09.749 ***** 2021-02-11T09:39:43.866 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2021-02-11T09:39:43.867 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => {"changed": false, "cmd": ["perldoc", "-l", "Amazon::S3"], "delta": "0:00:00.051305", "end": "2021-02-11 09:39:44.094721", "rc": 0, "start": "2021-02-11 09:39:44.043416", "stderr": "", "stderr_lines": [], "stdout": "/usr/local/share/perl/5.26.1/Amazon/S3.pm", "stdout_lines": ["/usr/local/share/perl/5.26.1/Amazon/S3.pm"]} 2021-02-11T09:39:43.903 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:43 +0000 (0:00:00.074) 0:02:09.824 ***** 2021-02-11T09:39:43.930 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2021-02-11T09:39:43.932 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:43.965 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:43 +0000 (0:00:00.061) 0:02:09.886 ***** 2021-02-11T09:39:43.971 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2021-02-11T09:39:43.972 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "size": 4096, "state": "directory", "uid": 1000} 2021-02-11T09:39:44.009 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:44 +0000 (0:00:00.043) 0:02:09.930 ***** 2021-02-11T09:39:44.015 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2021-02-11T09:39:44.017 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:44.052 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:44 +0000 (0:00:00.043) 0:02:09.973 ***** 2021-02-11T09:39:44.309 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2021-02-11T09:39:44.310 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "size": 73, "state": "file", "uid": 1000} 2021-02-11T09:39:44.345 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:44 +0000 (0:00:00.293) 0:02:10.266 ***** 2021-02-11T09:39:44.916 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2021-02-11T09:39:44.917 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => {"cache_update_time": 1613036065, "cache_updated": false, "changed": false} 2021-02-11T09:39:44.951 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:44 +0000 (0:00:00.605) 0:02:10.872 ***** 2021-02-11T09:39:44.953 INFO:teuthology.task.ansible.out: 2021-02-11T09:39:44.979 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2021-02-11T09:39:44.980 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:45.016 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:45 +0000 (0:00:00.064) 0:02:10.936 ***** 2021-02-11T09:39:45.305 INFO:teuthology.task.ansible.out: 2021-02-11T09:39:45.306 INFO:teuthology.task.ansible.out:TASK [testnode : Create the ntp.conf file.] ************************************ ok: [smithi098.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "size": 2767, "state": "file", "uid": 0} 2021-02-11T09:39:45.341 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:45 +0000 (0:00:00.325) 0:02:11.262 ***** 2021-02-11T09:39:45.358 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2021-02-11T09:39:45.358 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "cmd": ["/usr/bin/pip", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "", "stderr_lines": [], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages\nRequirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages", "Requirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2021-02-11T09:39:45.393 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:45 +0000 (0:00:00.052) 0:02:11.314 ***** 2021-02-11T09:39:45.400 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2021-02-11T09:39:45.401 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:45.437 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:45 +0000 (0:00:00.043) 0:02:11.357 ***** 2021-02-11T09:39:45.652 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2021-02-11T09:39:45.653 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "ntp", "state": "started", "status": {"ActiveEnterTimestamp": "Thu 2021-02-11 09:37:13 UTC", "ActiveEnterTimestampMonotonic": "28876232", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "-.mount systemd-journald.socket system.slice basic.target network.target sysinit.target systemd-tmpfiles-setup.service", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2021-02-11 09:37:07 UTC", "AssertTimestampMonotonic": "22408126", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2021-02-11 09:37:07 UTC", "ConditionTimestampMonotonic": "22408125", "ConfigurationDirectoryMode": "0755", "Conflicts": "systemd-timesyncd.service shutdown.target", "ControlGroup": "/system.slice/ntp.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "Documen 2021-02-11T09:39:45.655 INFO:teuthology.task.ansible.out:tation": "man:ntpd(8)", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1723", "ExecMainStartTimestamp": "Thu 2021-02-11 09:37:13 UTC", "ExecMainStartTimestampMonotonic": "28876185", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/ntp.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "ntp.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2021-02-11 09:37:07 UTC", "InactiveExitTimestampMonotonic": "22409632", "InvocationID": "848dde2a0b2f4dbbaa325a7160cbd379", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128029", "LimitNPROCSoft": "128029", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "Limi 2021-02-11T09:39:45.656 INFO:teuthology.task.ansible.out:tRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "128029", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1723", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "ntp.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice -.mount sysinit.target", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2021-02-11 09:37:13 UTC", "S 2021-02-11T09:39:45.657 INFO:teuthology.task.ansible.out:tateChangeTimestampMonotonic": "28876232", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "2", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2021-02-11 09:37:13 UTC", "WatchdogTimestampMonotonic": "28876230", "WatchdogUSec": "0"}} 2021-02-11T09:39:45.689 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:45 +0000 (0:00:00.252) 0:02:11.610 ***** 2021-02-11T09:39:45.715 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-11T09:39:45.717 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-02-11T09:39:45.749 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:45 +0000 (0:00:00.060) 0:02:11.670 ***** 2021-02-11T09:39:45.775 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:45.808 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:45 +0000 (0:00:00.059) 0:02:11.729 ***** 2021-02-11T09:39:45.834 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2021-02-11T09:39:45.835 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:45.868 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:45 +0000 (0:00:00.059) 0:02:11.789 ***** 2021-02-11T09:39:45.884 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2021-02-11T09:39:45.885 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false} 2021-02-11T09:39:45.920 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:45 +0000 (0:00:00.051) 0:02:11.840 ***** 2021-02-11T09:39:45.947 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ ok: [smithi094.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2021-02-11T09:39:45.981 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:45 +0000 (0:00:00.061) 0:02:11.902 ***** 2021-02-11T09:39:46.113 INFO:teuthology.task.ansible.out: 2021-02-11T09:39:46.115 INFO:teuthology.task.ansible.out:TASK [testnode : Check for presence of /etc/network/interfaces] **************** ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1613035992.3640413, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "ctime": 1613035992.3600414, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 746, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/plain", "mode": "0644", "mtime": 1613035992.3600414, "nlink": 1, "path": "/etc/network/interfaces", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 71, "uid": 0, "version": "3375002756", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} 2021-02-11T09:39:46.149 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:46 +0000 (0:00:00.168) 0:02:12.070 ***** 2021-02-11T09:39:46.549 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2021-02-11T09:39:46.551 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "msg": ""} 2021-02-11T09:39:46.589 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:46 +0000 (0:00:00.439) 0:02:12.509 ***** 2021-02-11T09:39:46.615 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2021-02-11T09:39:46.617 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-02-11T09:39:46.758 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:46 +0000 (0:00:00.169) 0:02:12.679 ***** 2021-02-11T09:39:46.765 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2021-02-11T09:39:46.766 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => {"cache_update_time": 1613036065, "cache_updated": false, "changed": false} 2021-02-11T09:39:46.803 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:46 +0000 (0:00:00.044) 0:02:12.724 ***** 2021-02-11T09:39:46.808 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2021-02-11T09:39:46.810 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:46.845 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:46 +0000 (0:00:00.041) 0:02:12.765 ***** 2021-02-11T09:39:46.852 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2021-02-11T09:39:46.852 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:46.888 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:46 +0000 (0:00:00.043) 0:02:12.809 ***** 2021-02-11T09:39:46.967 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2021-02-11T09:39:46.969 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-02-11T09:39:47.004 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:47 +0000 (0:00:00.116) 0:02:12.925 ***** 2021-02-11T09:39:47.021 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2021-02-11T09:39:47.022 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "size": 4096, "state": "directory", "uid": 1000} 2021-02-11T09:39:47.059 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:47 +0000 (0:00:00.054) 0:02:12.980 ***** 2021-02-11T09:39:47.128 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2021-02-11T09:39:47.129 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-02-11T09:39:47.166 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:47 +0000 (0:00:00.106) 0:02:13.086 ***** 2021-02-11T09:39:47.193 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:47.228 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:47 +0000 (0:00:00.062) 0:02:13.149 ***** 2021-02-11T09:39:47.276 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2021-02-11T09:39:47.277 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2021-02-11T09:39:47.288 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2021-02-11T09:39:47.289 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2021-02-11T09:39:47.299 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} 2021-02-11T09:39:47.335 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:47 +0000 (0:00:00.106) 0:02:13.256 ***** 2021-02-11T09:39:47.363 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2021-02-11T09:39:47.363 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "size": 73, "state": "file", "uid": 1000} 2021-02-11T09:39:47.400 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:47 +0000 (0:00:00.065) 0:02:13.321 ***** 2021-02-11T09:39:47.406 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2021-02-11T09:39:47.408 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:47.409 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:47.448 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:47 +0000 (0:00:00.047) 0:02:13.369 ***** 2021-02-11T09:39:47.475 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:47.511 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:47 +0000 (0:00:00.062) 0:02:13.431 ***** 2021-02-11T09:39:47.538 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2021-02-11T09:39:47.539 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:47.577 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:47 +0000 (0:00:00.066) 0:02:13.497 ***** 2021-02-11T09:39:47.603 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:47.640 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:47 +0000 (0:00:00.063) 0:02:13.561 ***** 2021-02-11T09:39:47.667 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:47.720 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:47 +0000 (0:00:00.079) 0:02:13.641 ***** 2021-02-11T09:39:47.747 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2021-02-11T09:39:47.749 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:47.785 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:47 +0000 (0:00:00.065) 0:02:13.706 ***** 2021-02-11T09:39:47.812 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-02-11T09:39:47.813 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:47.871 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:47 +0000 (0:00:00.084) 0:02:13.791 ***** 2021-02-11T09:39:47.898 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2021-02-11T09:39:47.899 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:47.939 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:47 +0000 (0:00:00.068) 0:02:13.860 ***** 2021-02-11T09:39:47.967 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-02-11T09:39:47.968 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:48.018 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:48 +0000 (0:00:00.079) 0:02:13.939 ***** 2021-02-11T09:39:48.172 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-02-11T09:39:51.578 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.011031", "end": "2021-02-11 09:39:48.142516", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-02-11 09:39:48.131485", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} TASK [testnode : Install packages via pip] ************************************* ok: [smithi098.front.sepia.ceph.com] => {"changed": false, "cmd": ["/usr/bin/pip", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "", "stderr_lines": [], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages\nRequirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages", "Requirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} Thursday 11 February 2021 09:39:48 +0000 (0:00:00.218) 0:02:14.157 ***** TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** ok: [smithi094.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.006065", "end": "2021-02-11 09:39:48.258998", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-02-11 09:39:48.252933", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} ok: [smithi094.front.sepia.ceph.com] => (ite 2021-02-11T09:39:51.579 INFO:teuthology.task.ansible.out:m=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.005460", "end": "2021-02-11 09:39:48.368645", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-02-11 09:39:48.363185", "stderr": "find: '/etc/ceph': No such file or directory", "stderr_lines": ["find: '/etc/ceph': No such file or directory"], "stdout": "", "stdout_lines": []} ok: [smithi094.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.007738", "end": "2021-02-11 09:39:48.481431", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-02-11 09:39:48.473693", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} Thursday 11 February 2021 09:39:48 +0000 (0:00:00.299) 0:02:14.457 ***** TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** Thursday 11 February 2021 09:39:48 +0000 (0:00:00.060) 0:02:14.518 ***** TASK [testnode : Touch /ceph-qa-ready] ***************************************** changed: [smithi094.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 0, "state": "file", "uid": 0} TASK [testnode : Purge resolvconf] ********************************************* ok: [smithi098.front.sepia.ceph.com] => {"changed": false} Thursday 11 February 2021 09:39:48 +0000 (0:00:00.219) 0:02:14.737 ***** TASK [testnode : Set interface] ************************************************ ok: [smithi098.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} Thursday 11 February 2021 09:39:48 +0000 (0:00:00.068) 0:02:14.806 ***** TASK [testnode : Check for presence of /etc/network/interfaces] **************** ok: [smithi098.front.sepia.ceph.com] => {"changed": false, "sta 2021-02-11T09:39:51.581 INFO:teuthology.task.ansible.out:t": {"atime": 1613035996.1357567, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "ctime": 1613035996.1317568, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 746, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/plain", "mode": "0644", "mtime": 1613035996.1317568, "nlink": 1, "path": "/etc/network/interfaces", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 71, "uid": 0, "version": "3375002756", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} Thursday 11 February 2021 09:39:49 +0000 (0:00:00.179) 0:02:14.985 ***** TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** ok: [smithi098.front.sepia.ceph.com] => {"changed": false, "msg": ""} Thursday 11 February 2021 09:39:49 +0000 (0:00:00.146) 0:02:15.131 ***** TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** ok: [smithi098.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} Thursday 11 February 2021 09:39:49 +0000 (0:00:00.060) 0:02:15.192 ***** TASK [testnode : ifdown and ifup] ********************************************** skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 09:39:49 +0000 (0:00:00.060) 0:02:15.252 ***** TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** changed: [smithi098.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} Thursday 11 February 2021 09:39:49 +0000 (0:00:00.163) 0:02:15.416 ***** TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* cha 2021-02-11T09:39:51.582 INFO:teuthology.task.ansible.out:nged: [smithi098.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} Thursday 11 February 2021 09:39:49 +0000 (0:00:00.167) 0:02:15.583 ***** TASK [testnode : Make sure ethtool is installed] ******************************* skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 09:39:49 +0000 (0:00:00.062) 0:02:15.645 ***** TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* skipping: [smithi098.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} skipping: [smithi098.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} skipping: [smithi098.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} skipping: [smithi098.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} skipping: [smithi098.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} Thursday 11 February 2021 09:39:49 +0000 (0:00:00.094) 0:02:15.740 ***** TASK [testnode : Define net_to_configure var] ********************************** skipping: [smithi098.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was Fa 2021-02-11T09:39:51.583 INFO:teuthology.task.ansible.out:lse"} skipping: [smithi098.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi098.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi098.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi098.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 09:39:49 +0000 (0:00:00.096) 0:02:15.837 ***** TASK [testnode : set_fact] ***************************************************** skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 09:39:49 +0000 (0:00:00 2021-02-11T09:39:51.585 INFO:teuthology.task.ansible.out:.064) 0:02:15.901 ***** TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 09:39:50 +0000 (0:00:00.061) 0:02:15.963 ***** TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 09:39:50 +0000 (0:00:00.063) 0:02:16.027 ***** TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 09:39:50 +0000 (0:00:00.061) 0:02:16.088 ***** TASK [testnode : Set MTU on Ubuntu] ******************************************** skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 09:39:50 +0000 (0:00:00.063) 0:02:16.152 ***** TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 09:39:50 +0000 (0:00:00.065) 0:02:16.217 ***** TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 09:39:50 +0000 (0:00:00.063) 0:02:16.280 ***** TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 09:39:50 +0000 (0:00:00.061) 0:02:16.342 ***** TASK [testno 2021-02-11T09:39:51.586 INFO:teuthology.task.ansible.out:de : List any leftover Ceph artifacts from previous jobs] ********** ok: [smithi098.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.008546", "end": "2021-02-11 09:39:50.794165", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-02-11 09:39:50.785619", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} ok: [smithi098.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.004576", "end": "2021-02-11 09:39:50.906152", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-02-11 09:39:50.901576", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} ok: [smithi098.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.005161", "end": "2021-02-11 09:39:51.019413", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-02-11 09:39:51.014252", "stderr": "find: '/etc/ceph': No such file or directory", "stderr_lines": ["find: '/etc/ceph': No such file or directory"], "stdout": "", "stdout_lines": []} ok: [smithi098.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.025682", "end": "2021-02-11 09:39:51.148690", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-02-11 09:39:51.123008", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} Thursday 11 February 2021 09:39:50 +0000 (0:00:00.547) 0:02:16.889 ***** TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** Thursday 11 February 2021 09:39:51 +0000 (0:00:00.073) 0:02:16.962 ***** TASK [ 2021-02-11T09:39:51.587 INFO:teuthology.task.ansible.out:testnode : Touch /ceph-qa-ready] ***************************************** changed: [smithi098.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 0, "state": "file", "uid": 0} PLAY [testnodes,senta,vossi,jenkins_slaves] ************************************ Thursday 11 February 2021 09:39:51 +0000 (0:00:00.273) 0:02:17.235 ***** Thursday 11 February 2021 09:39:51 +0000 (0:00:00.034) 0:02:17.270 ***** 2021-02-11T09:39:52.065 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi098.front.sepia.ceph.com] 2021-02-11T09:39:52.142 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:52 +0000 (0:00:00.792) 0:02:18.063 ***** 2021-02-11T09:39:52.153 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] 2021-02-11T09:39:52.197 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** ok: [smithi098.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "apt"}, "changed": false} 2021-02-11T09:39:52.232 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:52 +0000 (0:00:00.089) 0:02:18.153 ***** 2021-02-11T09:39:52.267 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:52 +0000 (0:00:00.035) 0:02:18.188 ***** 2021-02-11T09:39:52.273 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "apt"}, "changed": false} 2021-02-11T09:39:52.312 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:52 +0000 (0:00:00.044) 0:02:18.232 ***** 2021-02-11T09:39:52.318 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-02-11T09:39:52.319 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:52.357 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:52 +0000 (0:00:00.045) 0:02:18.278 ***** 2021-02-11T09:39:52.363 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:52.402 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:52 +0000 (0:00:00.045) 0:02:18.323 ***** 2021-02-11T09:39:52.409 INFO:teuthology.task.ansible.out: TASK [container-host : Including distro specific variables] ******************** 2021-02-11T09:39:52.409 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml) => {"ansible_facts": {"container_packages": ["docker.io"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml"} 2021-02-11T09:39:52.448 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:52 +0000 (0:00:00.045) 0:02:18.369 ***** 2021-02-11T09:39:52.454 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml) => {"ansible_facts": {"container_packages": ["docker.io"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml"} 2021-02-11T09:39:52.494 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:52 +0000 (0:00:00.045) 0:02:18.415 ***** 2021-02-11T09:39:53.295 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** 2021-02-11T09:39:53.296 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => {"cache_update_time": 1613036065, "cache_updated": false, "changed": false} 2021-02-11T09:39:53.349 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:53 +0000 (0:00:00.854) 0:02:19.269 ***** 2021-02-11T09:39:53.378 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-02-11T09:39:53.380 INFO:teuthology.task.ansible.out:skipping: [smithi098.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:53.417 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:53 +0000 (0:00:00.068) 0:02:19.338 ***** 2021-02-11T09:39:53.423 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** 2021-02-11T09:39:53.424 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"cache_update_time": 1613036022, "cache_updated": false, "changed": false} 2021-02-11T09:39:53.461 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:53 +0000 (0:00:00.043) 0:02:19.382 ***** 2021-02-11T09:39:53.467 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-02-11T09:39:53.468 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/docker/daemon.json"}, "changed": false} 2021-02-11T09:39:53.505 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:53 +0000 (0:00:00.044) 0:02:19.426 ***** 2021-02-11T09:39:53.511 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-02-11T09:39:53.512 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T09:39:53.552 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:53 +0000 (0:00:00.046) 0:02:19.473 ***** 2021-02-11T09:39:53.590 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-02-11T09:39:53.592 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/docker/daemon.json"}, "changed": false} 2021-02-11T09:39:53.627 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:53 +0000 (0:00:00.075) 0:02:19.548 ***** 2021-02-11T09:39:53.675 INFO:teuthology.task.ansible.out: TASK [container-host : Create container_mirror_cert_paths] ********************* 2021-02-11T09:39:53.676 INFO:teuthology.task.ansible.out:changed: [smithi098.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "size": 4096, "state": "directory", "uid": 0} 2021-02-11T09:39:53.781 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "size": 4096, "state": "directory", "uid": 0} 2021-02-11T09:39:53.781 INFO:teuthology.task.ansible.out:changed: [smithi098.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "size": 4096, "state": "directory", "uid": 0} 2021-02-11T09:39:53.822 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:53 +0000 (0:00:00.194) 0:02:19.742 ***** 2021-02-11T09:39:53.880 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "size": 4096, "state": "directory", "uid": 0} 2021-02-11T09:39:53.920 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:53 +0000 (0:00:00.097) 0:02:19.840 ***** 2021-02-11T09:39:54.163 INFO:teuthology.task.ansible.out: TASK [container-host : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert] *** 2021-02-11T09:39:54.165 INFO:teuthology.task.ansible.out:changed: [smithi098.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1613036393.8585846-104274508543144/source", "state": "file", "uid": 0} 2021-02-11T09:39:54.247 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1613036393.9610338-124141847167142/source", "state": "file", "uid": 0} 2021-02-11T09:39:54.466 INFO:teuthology.task.ansible.out:changed: [smithi098.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1613036394.1735692-97443043040056/source", "state": "file", "uid": 0} 2021-02-11T09:39:54.505 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:54 +0000 (0:00:00.585) 0:02:20.425 ***** 2021-02-11T09:39:54.536 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1613036394.2536948-88216072157263/source", "state": "file", "uid": 0} 2021-02-11T09:39:54.573 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:54 +0000 (0:00:00.068) 0:02:20.494 ***** 2021-02-11T09:39:57.993 INFO:teuthology.task.ansible.out: TASK [container-host : Install registries-conf-ctl] **************************** 2021-02-11T09:39:57.995 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => {"changed": true, "cmd": ["/usr/bin/pip2", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "", "stderr_lines": [], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-l5DILA-build\nCollecting docopt (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz\nCollecting toml (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl\nCollecting typing (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/3b/c0/e44213fcb799eac02881e2485724ba5b0914600bc9df6ed922e364fdc059/typing-3.7.4.3-py2-none-any.whl\nBuilding wheels for collected packages: docopt\n Running setup.py bdist_wheel for docopt: started\n Running setup.py bdist_wheel for docopt: finished with status 'done'\n Stored in directory: /root/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e\nSuccessfully built docopt\nInstalling collected packages: docopt, toml, typing, registries-conf-ctl\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2 typing-3.7.4.3\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-l5DILA-build", "Collecting docopt (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packag 2021-02-11T09:39:57.996 INFO:teuthology.task.ansible.out:es/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz", "Collecting toml (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl", "Collecting typing (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/3b/c0/e44213fcb799eac02881e2485724ba5b0914600bc9df6ed922e364fdc059/typing-3.7.4.3-py2-none-any.whl", "Building wheels for collected packages: docopt", " Running setup.py bdist_wheel for docopt: started", " Running setup.py bdist_wheel for docopt: finished with status 'done'", " Stored in directory: /root/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e", "Successfully built docopt", "Installing collected packages: docopt, toml, typing, registries-conf-ctl", " Running setup.py install for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2 typing-3.7.4.3"], "version": null, "virtualenv": null} 2021-02-11T09:39:58.031 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:58 +0000 (0:00:03.457) 0:02:23.951 ***** 2021-02-11T09:39:58.048 INFO:teuthology.task.ansible.out:changed: [smithi098.front.sepia.ceph.com] => {"changed": true, "cmd": ["/usr/bin/pip2", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "", "stderr_lines": [], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-ofwJOs-build\nCollecting docopt (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz\nCollecting toml (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl\nCollecting typing (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/3b/c0/e44213fcb799eac02881e2485724ba5b0914600bc9df6ed922e364fdc059/typing-3.7.4.3-py2-none-any.whl\nBuilding wheels for collected packages: docopt\n Running setup.py bdist_wheel for docopt: started\n Running setup.py bdist_wheel for docopt: finished with status 'done'\n Stored in directory: /root/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e\nSuccessfully built docopt\nInstalling collected packages: docopt, toml, typing, registries-conf-ctl\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2 typing-3.7.4.3\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-ofwJOs-build", "Collecting docopt (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packag 2021-02-11T09:39:58.049 INFO:teuthology.task.ansible.out:es/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz", "Collecting toml (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl", "Collecting typing (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/3b/c0/e44213fcb799eac02881e2485724ba5b0914600bc9df6ed922e364fdc059/typing-3.7.4.3-py2-none-any.whl", "Building wheels for collected packages: docopt", " Running setup.py bdist_wheel for docopt: started", " Running setup.py bdist_wheel for docopt: finished with status 'done'", " Stored in directory: /root/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e", "Successfully built docopt", "Installing collected packages: docopt, toml, typing, registries-conf-ctl", " Running setup.py install for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2 typing-3.7.4.3"], "version": null, "virtualenv": null} 2021-02-11T09:39:58.087 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:58 +0000 (0:00:00.056) 0:02:24.008 ***** 2021-02-11T09:39:58.176 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* 2021-02-11T09:39:58.178 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-02-11T09:39:58.217 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:58 +0000 (0:00:00.129) 0:02:24.137 ***** 2021-02-11T09:39:58.233 INFO:teuthology.task.ansible.out:ok: [smithi098.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-02-11T09:39:58.271 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:58 +0000 (0:00:00.054) 0:02:24.192 ***** 2021-02-11T09:39:58.572 INFO:teuthology.task.ansible.out: TASK [container-host : Create /etc/docker/daemon.json if necessary] ************ 2021-02-11T09:39:58.573 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => {"changed": true, "checksum": "bf21a9e8fbc5a3846fb05b4fa0859e0917b2202f", "dest": "/etc/docker/daemon.json", "gid": 0, "group": "root", "md5sum": "99914b932bd37a50b983c5e7c90ae93b", "mode": "0644", "owner": "root", "size": 2, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1613036398.2481647-245836788130556/source", "state": "file", "uid": 0} 2021-02-11T09:39:58.628 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:58 +0000 (0:00:00.357) 0:02:24.549 ***** 2021-02-11T09:39:58.667 INFO:teuthology.task.ansible.out:changed: [smithi098.front.sepia.ceph.com] => {"changed": true, "checksum": "bf21a9e8fbc5a3846fb05b4fa0859e0917b2202f", "dest": "/etc/docker/daemon.json", "gid": 0, "group": "root", "md5sum": "99914b932bd37a50b983c5e7c90ae93b", "mode": "0644", "owner": "root", "size": 2, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1613036398.3378842-237216168626519/source", "state": "file", "uid": 0} 2021-02-11T09:39:58.710 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:58 +0000 (0:00:00.081) 0:02:24.630 ***** 2021-02-11T09:39:58.987 INFO:teuthology.task.ansible.out: TASK [container-host : Add local docker.io registry mirror] ******************** 2021-02-11T09:39:58.989 INFO:teuthology.task.ansible.out:changed: [smithi098.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.135064", "end": "2021-02-11 09:39:59.203465", "rc": 0, "start": "2021-02-11 09:39:59.068401", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-11T09:39:59.037 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:59 +0000 (0:00:00.327) 0:02:24.958 ***** 2021-02-11T09:39:59.181 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.423926", "end": "2021-02-11 09:39:59.168035", "rc": 0, "start": "2021-02-11 09:39:58.744109", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-11T09:39:59.219 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 09:39:59 +0000 (0:00:00.181) 0:02:25.140 ***** 2021-02-11T09:40:02.489 INFO:teuthology.task.ansible.out: TASK [container-host : Restart docker service] ********************************* 2021-02-11T09:40:02.490 INFO:teuthology.task.ansible.out:changed: [smithi098.front.sepia.ceph.com] => {"changed": true, "name": "docker", "state": "started", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "network-online.target containerd.service basic.target sysinit.target systemd-journald.socket firewalld.service system.slice docker.socket", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ConsistsOf": "docker.socket", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "yes", "DelegateControllers": "cpu cpuacct io blkio memory devices pids", "Description": "Docker Application Container Engine", "DevicePolicy": "auto", "Documentation": "https://docs.docker.com", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMa 2021-02-11T09:40:02.491 INFO:teuthology.task.ansible.out:inStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -s HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/bin/dockerd ; argv[]=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/docker.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "docker.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "process", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "infinity", "LimitNOFILESoft": "infinity", "LimitNPROC": "infinity", "LimitNPROCSoft": "infinity", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "12802 2021-02-11T09:40:02.493 INFO:teuthology.task.ansible.out:9", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "docker.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "main", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target docker.socket", "Restart": "always", "RestartUSec": "2s", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "3", "StartLimitIntervalUSec": "1min", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2021-02-11 09:39:00 UTC", "StateChangeTimestampMonotonic": "135714837", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "S 2021-02-11T09:40:02.494 INFO:teuthology.task.ansible.out:uccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "infinity", "TimeoutStartUSec": "infinity", "TimeoutStopUSec": "infinity", "TimerSlackNSec": "50000", "Transient": "no", "TriggeredBy": "docker.socket", "Type": "notify", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "containerd.service network-online.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-02-11T09:40:02.613 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => {"changed": true, "name": "docker", "state": "started", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "docker.socket system.slice sysinit.target firewalld.service containerd.service basic.target systemd-journald.socket network-online.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ConsistsOf": "docker.socket", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "yes", "DelegateControllers": "cpu cpuacct io blkio memory devices pids", "Description": "Docker Application Container Engine", "DevicePolicy": "auto", "Documentation": "https://docs.docker.com", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMa 2021-02-11T09:40:02.615 INFO:teuthology.task.ansible.out:inStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -s HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/bin/dockerd ; argv[]=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/docker.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "docker.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "process", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "infinity", "LimitNOFILESoft": "infinity", "LimitNPROC": "infinity", "LimitNPROCSoft": "infinity", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "12802 2021-02-11T09:40:02.616 INFO:teuthology.task.ansible.out:9", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "docker.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "main", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "docker.socket system.slice sysinit.target", "Restart": "always", "RestartUSec": "2s", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "3", "StartLimitIntervalUSec": "1min", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2021-02-11 09:38:58 UTC", "StateChangeTimestampMonotonic": "142260294", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "S 2021-02-11T09:40:02.618 INFO:teuthology.task.ansible.out:uccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "infinity", "TimeoutStartUSec": "infinity", "TimeoutStopUSec": "infinity", "TimerSlackNSec": "50000", "Transient": "no", "TriggeredBy": "docker.socket", "Type": "notify", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "network-online.target containerd.service", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-02-11T09:40:02.709 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2021-02-11T09:40:02.713 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** skipping: no hosts matched 2021-02-11T09:40:02.716 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** skipping: no hosts matched PLAY RECAP ********************************************************************* 2021-02-11T09:40:02.718 INFO:teuthology.task.ansible.out:smithi094.front.sepia.ceph.com : ok=126 changed=38 unreachable=0 failed=0 skipped=184 rescued=0 ignored=0 smithi098.front.sepia.ceph.com : ok=126 changed=38 unreachable=0 failed=0 skipped=184 rescued=0 ignored=0 Thursday 11 February 2021 09:40:02 +0000 (0:00:03.497) 0:02:28.638 ***** =============================================================================== users : Create all admin users with sudo access. ----------------------- 17.67s users : Update authorized_keys using the keys repo --------------------- 14.21s testnode : Install packages --------------------------------------------- 7.05s testnode : Ensure packages are not present. ----------------------------- 6.98s testnode : Zap all non-root disks --------------------------------------- 6.53s container-host : Restart docker service --------------------------------- 3.50s container-host : Install registries-conf-ctl ---------------------------- 3.46s users : Remove revoked users -------------------------------------------- 2.31s users : Clone the keys repo --------------------------------------------- 2.31s Gathering Facts --------------------------------------------------------- 1.85s users : Update authorized_keys for each user with literal keys ---------- 1.58s testnode : Update apt cache. -------------------------------------------- 1.04s testnode : Upgrade packages --------------------------------------------- 0.96s testnode : Make sure apt dependencies are installed --------------------- 0.92s testnode : Install python-apt ------------------------------------------- 0.92s container-host : Install container packages ----------------------------- 0.85s testnode : Create xfs filesystem on /dev/vg_nvme/lv_5 ------------------- 0.82s Gathering Facts --------------------------------------------------------- 0.79s Gathering Facts --------------------------------------------------------- 0.77s ansible-managed : Create the cephlab_sudo sudoers 2021-02-11T09:40:02.719 INFO:teuthology.task.ansible.out:.d file. --------------- 0.75s 2021-02-11T09:40:03.027 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi094.front.sepia.ceph.com'), Remote(name='ubuntu@smithi098.front.sepia.ceph.com')] 2021-02-11T09:40:03.028 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-11T09:40:03.030 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-02-11T09:40:03.109 DEBUG:teuthology.orchestra.run.smithi094:> true 2021-02-11T09:40:03.307 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-11T09:40:03.308 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-02-11T09:40:03.379 DEBUG:teuthology.orchestra.run.smithi098:> true 2021-02-11T09:40:03.571 INFO:teuthology.run_tasks:Running task clock... 2021-02-11T09:40:03.594 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2021-02-11T09:40:03.595 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-02-11T09:40:03.596 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-02-11T09:40:03.666 INFO:teuthology.orchestra.run.smithi094.stdout:11 Feb 09:40:03 ntpd[8197]: ntpd 4.2.8p10@1.3728-o (1): Starting 2021-02-11T09:40:03.666 INFO:teuthology.orchestra.run.smithi094.stdout:11 Feb 09:40:03 ntpd[8197]: Command line: ntpd -gq 2021-02-11T09:40:03.666 INFO:teuthology.orchestra.run.smithi094.stdout:11 Feb 09:40:03 ntpd[8197]: proto: precision = 0.123 usec (-23) 2021-02-11T09:40:03.667 INFO:teuthology.orchestra.run.smithi094.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2021-02-11T09:40:03.667 INFO:teuthology.orchestra.run.smithi094.stderr:restrict ::: KOD does nothing without LIMITED. 2021-02-11T09:40:03.668 INFO:teuthology.orchestra.run.smithi094.stdout:11 Feb 09:40:03 ntpd[8197]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2021-02-11T09:40:03.668 INFO:teuthology.orchestra.run.smithi094.stdout:11 Feb 09:40:03 ntpd[8197]: restrict ::: KOD does nothing without LIMITED. 2021-02-11T09:40:03.668 INFO:teuthology.orchestra.run.smithi094.stdout:11 Feb 09:40:03 ntpd[8197]: Listen and drop on 0 v6wildcard [::]:123 2021-02-11T09:40:03.668 INFO:teuthology.orchestra.run.smithi094.stdout:11 Feb 09:40:03 ntpd[8197]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2021-02-11T09:40:03.668 INFO:teuthology.orchestra.run.smithi094.stdout:11 Feb 09:40:03 ntpd[8197]: Listen normally on 2 lo 127.0.0.1:123 2021-02-11T09:40:03.668 INFO:teuthology.orchestra.run.smithi094.stdout:11 Feb 09:40:03 ntpd[8197]: Listen normally on 3 enp3s0f1 172.21.15.94:123 2021-02-11T09:40:03.668 INFO:teuthology.orchestra.run.smithi094.stdout:11 Feb 09:40:03 ntpd[8197]: Listen normally on 4 lo [::1]:123 2021-02-11T09:40:03.668 INFO:teuthology.orchestra.run.smithi094.stdout:11 Feb 09:40:03 ntpd[8197]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d763%5]:123 2021-02-11T09:40:03.668 INFO:teuthology.orchestra.run.smithi094.stdout:11 Feb 09:40:03 ntpd[8197]: Listening on routing socket on fd #22 for interface updates 2021-02-11T09:40:04.666 INFO:teuthology.orchestra.run.smithi094.stderr:11 Feb 09:40:04 ntpd[8197]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T09:40:04.666 INFO:teuthology.orchestra.run.smithi094.stderr:11 Feb 09:40:04 ntpd[8197]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T09:40:04.667 INFO:teuthology.orchestra.run.smithi094.stderr:11 Feb 09:40:04 ntpd[8197]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T09:40:04.667 INFO:teuthology.orchestra.run.smithi094.stderr:11 Feb 09:40:04 ntpd[8197]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T09:40:05.666 INFO:teuthology.orchestra.run.smithi094.stderr:11 Feb 09:40:05 ntpd[8197]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T09:40:05.666 INFO:teuthology.orchestra.run.smithi094.stderr:11 Feb 09:40:05 ntpd[8197]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T09:40:05.667 INFO:teuthology.orchestra.run.smithi094.stderr:11 Feb 09:40:05 ntpd[8197]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T09:40:05.667 INFO:teuthology.orchestra.run.smithi094.stderr:11 Feb 09:40:05 ntpd[8197]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T09:40:06.667 INFO:teuthology.orchestra.run.smithi094.stderr:11 Feb 09:40:06 ntpd[8197]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T09:40:06.667 INFO:teuthology.orchestra.run.smithi094.stderr:11 Feb 09:40:06 ntpd[8197]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T09:40:06.667 INFO:teuthology.orchestra.run.smithi094.stderr:11 Feb 09:40:06 ntpd[8197]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T09:40:06.667 INFO:teuthology.orchestra.run.smithi094.stderr:11 Feb 09:40:06 ntpd[8197]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T09:40:06.667 INFO:teuthology.orchestra.run.smithi094.stderr:11 Feb 09:40:06 ntpd[8197]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T09:40:06.667 INFO:teuthology.orchestra.run.smithi094.stderr:11 Feb 09:40:06 ntpd[8197]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T09:40:06.667 INFO:teuthology.orchestra.run.smithi094.stderr:11 Feb 09:40:06 ntpd[8197]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T09:40:06.667 INFO:teuthology.orchestra.run.smithi094.stderr:11 Feb 09:40:06 ntpd[8197]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T09:40:07.666 INFO:teuthology.orchestra.run.smithi094.stderr:11 Feb 09:40:07 ntpd[8197]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T09:40:07.666 INFO:teuthology.orchestra.run.smithi094.stderr:11 Feb 09:40:07 ntpd[8197]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T09:40:07.666 INFO:teuthology.orchestra.run.smithi094.stderr:11 Feb 09:40:07 ntpd[8197]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T09:40:07.667 INFO:teuthology.orchestra.run.smithi094.stderr:11 Feb 09:40:07 ntpd[8197]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T09:40:08.667 INFO:teuthology.orchestra.run.smithi094.stderr:11 Feb 09:40:08 ntpd[8197]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T09:40:08.667 INFO:teuthology.orchestra.run.smithi094.stderr:11 Feb 09:40:08 ntpd[8197]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T09:40:08.667 INFO:teuthology.orchestra.run.smithi094.stderr:11 Feb 09:40:08 ntpd[8197]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T09:40:08.667 INFO:teuthology.orchestra.run.smithi094.stderr:11 Feb 09:40:08 ntpd[8197]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T09:40:08.667 INFO:teuthology.orchestra.run.smithi094.stderr:11 Feb 09:40:08 ntpd[8197]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T09:40:08.667 INFO:teuthology.orchestra.run.smithi094.stderr:11 Feb 09:40:08 ntpd[8197]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T09:40:08.667 INFO:teuthology.orchestra.run.smithi094.stderr:11 Feb 09:40:08 ntpd[8197]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T09:40:08.667 INFO:teuthology.orchestra.run.smithi094.stderr:11 Feb 09:40:08 ntpd[8197]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T09:40:08.667 INFO:teuthology.orchestra.run.smithi094.stderr:11 Feb 09:40:08 ntpd[8197]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T09:40:08.667 INFO:teuthology.orchestra.run.smithi094.stderr:11 Feb 09:40:08 ntpd[8197]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T09:40:08.668 INFO:teuthology.orchestra.run.smithi094.stderr:11 Feb 09:40:08 ntpd[8197]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T09:40:08.668 INFO:teuthology.orchestra.run.smithi094.stderr:11 Feb 09:40:08 ntpd[8197]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T09:40:09.670 INFO:teuthology.orchestra.run.smithi094.stderr:11 Feb 09:40:09 ntpd[8197]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T09:40:09.670 INFO:teuthology.orchestra.run.smithi094.stderr:11 Feb 09:40:09 ntpd[8197]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T09:40:09.671 INFO:teuthology.orchestra.run.smithi094.stderr:11 Feb 09:40:09 ntpd[8197]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T09:40:09.671 INFO:teuthology.orchestra.run.smithi094.stderr:11 Feb 09:40:09 ntpd[8197]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T09:40:10.667 INFO:teuthology.orchestra.run.smithi094.stderr:11 Feb 09:40:10 ntpd[8197]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T09:40:10.667 INFO:teuthology.orchestra.run.smithi094.stderr:11 Feb 09:40:10 ntpd[8197]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T09:40:10.667 INFO:teuthology.orchestra.run.smithi094.stderr:11 Feb 09:40:10 ntpd[8197]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T09:40:10.668 INFO:teuthology.orchestra.run.smithi094.stderr:11 Feb 09:40:10 ntpd[8197]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T09:40:10.668 INFO:teuthology.orchestra.run.smithi094.stderr:11 Feb 09:40:10 ntpd[8197]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T09:40:10.668 INFO:teuthology.orchestra.run.smithi094.stderr:11 Feb 09:40:10 ntpd[8197]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T09:40:10.668 INFO:teuthology.orchestra.run.smithi094.stderr:11 Feb 09:40:10 ntpd[8197]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T09:40:10.668 INFO:teuthology.orchestra.run.smithi094.stderr:11 Feb 09:40:10 ntpd[8197]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T09:40:10.668 INFO:teuthology.orchestra.run.smithi094.stderr:11 Feb 09:40:10 ntpd[8197]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T09:40:10.668 INFO:teuthology.orchestra.run.smithi094.stderr:11 Feb 09:40:10 ntpd[8197]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T09:40:10.668 INFO:teuthology.orchestra.run.smithi094.stderr:11 Feb 09:40:10 ntpd[8197]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T09:40:10.668 INFO:teuthology.orchestra.run.smithi094.stderr:11 Feb 09:40:10 ntpd[8197]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T09:40:10.668 INFO:teuthology.orchestra.run.smithi094.stderr:11 Feb 09:40:10 ntpd[8197]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2021-02-11T09:40:10.668 INFO:teuthology.orchestra.run.smithi094.stderr:11 Feb 09:40:10 ntpd[8197]: can't open /var/log/ntpstats/loopstats.20210211: Permission denied 2021-02-11T09:40:10.668 INFO:teuthology.orchestra.run.smithi094.stdout:11 Feb 09:40:10 ntpd[8197]: ntpd: time slew -0.003325 s 2021-02-11T09:40:10.668 INFO:teuthology.orchestra.run.smithi094.stdout:ntpd: time slew -0.003325s 2021-02-11T09:40:10.786 INFO:teuthology.orchestra.run.smithi094.stdout: remote refid st t when poll reach delay offset jitter 2021-02-11T09:40:10.787 INFO:teuthology.orchestra.run.smithi094.stdout:============================================================================== 2021-02-11T09:40:10.787 INFO:teuthology.orchestra.run.smithi094.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-02-11T09:40:10.787 INFO:teuthology.orchestra.run.smithi094.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-02-11T09:40:10.787 INFO:teuthology.orchestra.run.smithi094.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-02-11T09:40:10.787 INFO:teuthology.orchestra.run.smithi094.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-02-11T09:40:10.787 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-02-11T09:40:10.787 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-02-11T09:40:10.839 INFO:teuthology.orchestra.run.smithi098.stdout:11 Feb 09:40:11 ntpd[8234]: ntpd 4.2.8p10@1.3728-o (1): Starting 2021-02-11T09:40:10.839 INFO:teuthology.orchestra.run.smithi098.stdout:11 Feb 09:40:11 ntpd[8234]: Command line: ntpd -gq 2021-02-11T09:40:10.841 INFO:teuthology.orchestra.run.smithi098.stdout:11 Feb 09:40:11 ntpd[8234]: proto: precision = 0.097 usec (-23) 2021-02-11T09:40:10.841 INFO:teuthology.orchestra.run.smithi098.stdout:11 Feb 09:40:11 ntpd[8234]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2021-02-11T09:40:10.841 INFO:teuthology.orchestra.run.smithi098.stdout:11 Feb 09:40:11 ntpd[8234]: restrict ::: KOD does nothing without LIMITED. 2021-02-11T09:40:10.841 INFO:teuthology.orchestra.run.smithi098.stdout:11 Feb 09:40:11 ntpd[8234]: Listen and drop on 0 v6wildcard [::]:123 2021-02-11T09:40:10.841 INFO:teuthology.orchestra.run.smithi098.stdout:11 Feb 09:40:11 ntpd[8234]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2021-02-11T09:40:10.841 INFO:teuthology.orchestra.run.smithi098.stdout:11 Feb 09:40:11 ntpd[8234]: Listen normally on 2 lo 127.0.0.1:123 2021-02-11T09:40:10.841 INFO:teuthology.orchestra.run.smithi098.stdout:11 Feb 09:40:11 ntpd[8234]: Listen normally on 3 enp3s0f1 172.21.15.98:123 2021-02-11T09:40:10.841 INFO:teuthology.orchestra.run.smithi098.stdout:11 Feb 09:40:11 ntpd[8234]: Listen normally on 4 lo [::1]:123 2021-02-11T09:40:10.841 INFO:teuthology.orchestra.run.smithi098.stdout:11 Feb 09:40:11 ntpd[8234]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d26b%5]:123 2021-02-11T09:40:10.841 INFO:teuthology.orchestra.run.smithi098.stdout:11 Feb 09:40:11 ntpd[8234]: Listening on routing socket on fd #22 for interface updates 2021-02-11T09:40:10.841 INFO:teuthology.orchestra.run.smithi098.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2021-02-11T09:40:10.842 INFO:teuthology.orchestra.run.smithi098.stderr:restrict ::: KOD does nothing without LIMITED. 2021-02-11T09:40:11.840 INFO:teuthology.orchestra.run.smithi098.stderr:11 Feb 09:40:12 ntpd[8234]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T09:40:11.841 INFO:teuthology.orchestra.run.smithi098.stderr:11 Feb 09:40:12 ntpd[8234]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T09:40:11.841 INFO:teuthology.orchestra.run.smithi098.stderr:11 Feb 09:40:12 ntpd[8234]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T09:40:11.841 INFO:teuthology.orchestra.run.smithi098.stderr:11 Feb 09:40:12 ntpd[8234]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T09:40:12.841 INFO:teuthology.orchestra.run.smithi098.stderr:11 Feb 09:40:13 ntpd[8234]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T09:40:12.841 INFO:teuthology.orchestra.run.smithi098.stderr:11 Feb 09:40:13 ntpd[8234]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T09:40:12.841 INFO:teuthology.orchestra.run.smithi098.stderr:11 Feb 09:40:13 ntpd[8234]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T09:40:12.841 INFO:teuthology.orchestra.run.smithi098.stderr:11 Feb 09:40:13 ntpd[8234]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T09:40:13.842 INFO:teuthology.orchestra.run.smithi098.stderr:11 Feb 09:40:14 ntpd[8234]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T09:40:13.842 INFO:teuthology.orchestra.run.smithi098.stderr:11 Feb 09:40:14 ntpd[8234]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T09:40:13.842 INFO:teuthology.orchestra.run.smithi098.stderr:11 Feb 09:40:14 ntpd[8234]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T09:40:13.843 INFO:teuthology.orchestra.run.smithi098.stderr:11 Feb 09:40:14 ntpd[8234]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T09:40:13.843 INFO:teuthology.orchestra.run.smithi098.stderr:11 Feb 09:40:14 ntpd[8234]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T09:40:13.843 INFO:teuthology.orchestra.run.smithi098.stderr:11 Feb 09:40:14 ntpd[8234]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T09:40:13.843 INFO:teuthology.orchestra.run.smithi098.stderr:11 Feb 09:40:14 ntpd[8234]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T09:40:13.843 INFO:teuthology.orchestra.run.smithi098.stderr:11 Feb 09:40:14 ntpd[8234]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T09:40:14.842 INFO:teuthology.orchestra.run.smithi098.stderr:11 Feb 09:40:15 ntpd[8234]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T09:40:14.842 INFO:teuthology.orchestra.run.smithi098.stderr:11 Feb 09:40:15 ntpd[8234]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T09:40:14.843 INFO:teuthology.orchestra.run.smithi098.stderr:11 Feb 09:40:15 ntpd[8234]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T09:40:14.843 INFO:teuthology.orchestra.run.smithi098.stderr:11 Feb 09:40:15 ntpd[8234]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T09:40:14.843 INFO:teuthology.orchestra.run.smithi098.stderr:11 Feb 09:40:15 ntpd[8234]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T09:40:14.843 INFO:teuthology.orchestra.run.smithi098.stderr:11 Feb 09:40:15 ntpd[8234]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T09:40:14.843 INFO:teuthology.orchestra.run.smithi098.stderr:11 Feb 09:40:15 ntpd[8234]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T09:40:14.843 INFO:teuthology.orchestra.run.smithi098.stderr:11 Feb 09:40:15 ntpd[8234]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T09:40:15.842 INFO:teuthology.orchestra.run.smithi098.stderr:11 Feb 09:40:16 ntpd[8234]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T09:40:15.843 INFO:teuthology.orchestra.run.smithi098.stderr:11 Feb 09:40:16 ntpd[8234]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T09:40:15.843 INFO:teuthology.orchestra.run.smithi098.stderr:11 Feb 09:40:16 ntpd[8234]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T09:40:15.843 INFO:teuthology.orchestra.run.smithi098.stderr:11 Feb 09:40:16 ntpd[8234]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T09:40:15.843 INFO:teuthology.orchestra.run.smithi098.stderr:11 Feb 09:40:16 ntpd[8234]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T09:40:15.843 INFO:teuthology.orchestra.run.smithi098.stderr:11 Feb 09:40:16 ntpd[8234]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T09:40:15.843 INFO:teuthology.orchestra.run.smithi098.stderr:11 Feb 09:40:16 ntpd[8234]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T09:40:15.843 INFO:teuthology.orchestra.run.smithi098.stderr:11 Feb 09:40:16 ntpd[8234]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T09:40:16.843 INFO:teuthology.orchestra.run.smithi098.stderr:11 Feb 09:40:17 ntpd[8234]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T09:40:16.843 INFO:teuthology.orchestra.run.smithi098.stderr:11 Feb 09:40:17 ntpd[8234]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T09:40:16.843 INFO:teuthology.orchestra.run.smithi098.stderr:11 Feb 09:40:17 ntpd[8234]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T09:40:16.843 INFO:teuthology.orchestra.run.smithi098.stderr:11 Feb 09:40:17 ntpd[8234]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T09:40:16.843 INFO:teuthology.orchestra.run.smithi098.stderr:11 Feb 09:40:17 ntpd[8234]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T09:40:16.843 INFO:teuthology.orchestra.run.smithi098.stderr:11 Feb 09:40:17 ntpd[8234]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T09:40:16.843 INFO:teuthology.orchestra.run.smithi098.stderr:11 Feb 09:40:17 ntpd[8234]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T09:40:16.844 INFO:teuthology.orchestra.run.smithi098.stderr:11 Feb 09:40:17 ntpd[8234]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T09:40:17.843 INFO:teuthology.orchestra.run.smithi098.stderr:11 Feb 09:40:18 ntpd[8234]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T09:40:17.844 INFO:teuthology.orchestra.run.smithi098.stderr:11 Feb 09:40:18 ntpd[8234]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T09:40:17.844 INFO:teuthology.orchestra.run.smithi098.stderr:11 Feb 09:40:18 ntpd[8234]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T09:40:17.844 INFO:teuthology.orchestra.run.smithi098.stderr:11 Feb 09:40:18 ntpd[8234]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T09:40:17.844 INFO:teuthology.orchestra.run.smithi098.stderr:11 Feb 09:40:17 ntpd[8234]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2021-02-11T09:40:17.844 INFO:teuthology.orchestra.run.smithi098.stderr:11 Feb 09:40:17 ntpd[8234]: can't open /var/log/ntpstats/loopstats.20210211: Permission denied 2021-02-11T09:40:17.844 INFO:teuthology.orchestra.run.smithi098.stdout:11 Feb 09:40:17 ntpd[8234]: ntpd: time set -0.231378 s 2021-02-11T09:40:17.844 INFO:teuthology.orchestra.run.smithi098.stdout:ntpd: time set -0.231378s 2021-02-11T09:40:17.912 INFO:teuthology.orchestra.run.smithi098.stdout: remote refid st t when poll reach delay offset jitter 2021-02-11T09:40:17.912 INFO:teuthology.orchestra.run.smithi098.stdout:============================================================================== 2021-02-11T09:40:17.912 INFO:teuthology.orchestra.run.smithi098.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-02-11T09:40:17.912 INFO:teuthology.orchestra.run.smithi098.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-02-11T09:40:17.912 INFO:teuthology.orchestra.run.smithi098.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-02-11T09:40:17.912 INFO:teuthology.orchestra.run.smithi098.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-02-11T09:40:17.913 INFO:teuthology.run_tasks:Running task install... 2021-02-11T09:40:17.923 DEBUG:teuthology.task.install:project ceph 2021-02-11T09:40:17.923 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': '5e5bb33c614118d912ca83c2e5f2be67b3acc04f'}} 2021-02-11T09:40:17.923 DEBUG:teuthology.task.install:config {'sha1': '5e5bb33c614118d912ca83c2e5f2be67b3acc04f'} 2021-02-11T09:40:17.924 INFO:teuthology.task.install:Using flavor: basic 2021-02-11T09:40:17.939 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2021-02-11T09:40:17.939 INFO:teuthology.task.install:extra packages: [] 2021-02-11T09:40:17.939 DEBUG:teuthology.orchestra.run.smithi094:> sudo lsb_release -is 2021-02-11T09:40:17.992 INFO:teuthology.orchestra.run.smithi094.stdout:Ubuntu 2021-02-11T09:40:17.997 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-02-11T09:40:17.997 DEBUG:teuthology.orchestra.run.smithi098:> sudo lsb_release -is 2021-02-11T09:40:17.998 DEBUG:teuthology.orchestra.run.smithi094:> sudo apt-key list | grep Ceph 2021-02-11T09:40:18.049 INFO:teuthology.orchestra.run.smithi098.stdout:Ubuntu 2021-02-11T09:40:18.054 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-02-11T09:40:18.054 DEBUG:teuthology.orchestra.run.smithi098:> sudo apt-key list | grep Ceph 2021-02-11T09:40:18.089 INFO:teuthology.orchestra.run.smithi094.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2021-02-11T09:40:18.142 INFO:teuthology.orchestra.run.smithi098.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2021-02-11T09:40:18.145 INFO:teuthology.orchestra.run.smithi094.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2021-02-11T09:40:18.145 INFO:teuthology.orchestra.run.smithi094.stdout:uid [ unknown] Ceph.com (release key) 2021-02-11T09:40:18.146 DEBUG:teuthology.orchestra.run.smithi094:> sudo lsb_release -is 2021-02-11T09:40:18.194 INFO:teuthology.orchestra.run.smithi094.stdout:Ubuntu 2021-02-11T09:40:18.199 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-02-11T09:40:18.199 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2021-02-11T09:40:18.200 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=5e5bb33c614118d912ca83c2e5f2be67b3acc04f 2021-02-11T09:40:18.208 INFO:teuthology.orchestra.run.smithi098.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2021-02-11T09:40:18.208 INFO:teuthology.orchestra.run.smithi098.stdout:uid [ unknown] Ceph.com (release key) 2021-02-11T09:40:18.209 DEBUG:teuthology.orchestra.run.smithi098:> sudo lsb_release -is 2021-02-11T09:40:18.259 INFO:teuthology.orchestra.run.smithi098.stdout:Ubuntu 2021-02-11T09:40:18.264 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-02-11T09:40:18.264 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2021-02-11T09:40:18.265 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=5e5bb33c614118d912ca83c2e5f2be67b3acc04f 2021-02-11T09:40:18.383 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default/ 2021-02-11T09:40:18.384 INFO:teuthology.task.install.deb:Package version is 17.0.0-735-g5e5bb33c-1bionic 2021-02-11T09:40:18.436 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default/ 2021-02-11T09:40:18.436 INFO:teuthology.task.install.deb:Package version is 17.0.0-735-g5e5bb33c-1bionic 2021-02-11T09:40:18.516 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2021-02-11T09:40:18.517 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2021-02-11T09:40:18.532 DEBUG:teuthology.orchestra.run.smithi094:> sudo apt-get update 2021-02-11T09:40:18.566 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2021-02-11T09:40:18.566 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2021-02-11T09:40:18.582 DEBUG:teuthology.orchestra.run.smithi098:> sudo apt-get update 2021-02-11T09:40:18.793 INFO:teuthology.orchestra.run.smithi094.stdout:Ign:1 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic InRelease 2021-02-11T09:40:24.140 INFO:teuthology.orchestra.run.smithi098.stdout:Ign:1 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic InRelease 2021-02-11T09:40:24.140 INFO:teuthology.orchestra.run.smithi098.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic Release [7,681 B] 2021-02-11T09:40:24.140 INFO:teuthology.orchestra.run.smithi098.stdout:Ign:3 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic Release.gpg 2021-02-11T09:40:24.140 INFO:teuthology.orchestra.run.smithi098.stdout:Hit:4 http://security.ubuntu.com/ubuntu bionic-security InRelease 2021-02-11T09:40:24.140 INFO:teuthology.orchestra.run.smithi098.stdout:Hit:5 http://archive.ubuntu.com/ubuntu bionic InRelease 2021-02-11T09:40:24.140 INFO:teuthology.orchestra.run.smithi098.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 Packages [16.6 kB] 2021-02-11T09:40:24.140 INFO:teuthology.orchestra.run.smithi098.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main i386 Packages [3,096 B] 2021-02-11T09:40:24.141 INFO:teuthology.orchestra.run.smithi098.stdout:Hit:8 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2021-02-11T09:40:24.141 INFO:teuthology.orchestra.run.smithi098.stdout:Hit:9 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2021-02-11T09:40:24.141 INFO:teuthology.orchestra.run.smithi098.stdout:Fetched 27.3 kB in 1s (39.4 kB/s) 2021-02-11T09:40:24.141 INFO:teuthology.orchestra.run.smithi098.stdout:Reading package lists... 2021-02-11T09:40:24.141 INFO:teuthology.orchestra.run.smithi094.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic Release [7,681 B] 2021-02-11T09:40:24.141 INFO:teuthology.orchestra.run.smithi094.stdout:Hit:3 http://security.ubuntu.com/ubuntu bionic-security InRelease 2021-02-11T09:40:24.141 INFO:teuthology.orchestra.run.smithi094.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic InRelease 2021-02-11T09:40:24.141 INFO:teuthology.orchestra.run.smithi094.stdout:Ign:5 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic Release.gpg 2021-02-11T09:40:24.141 INFO:teuthology.orchestra.run.smithi094.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 Packages [16.6 kB] 2021-02-11T09:40:24.141 INFO:teuthology.orchestra.run.smithi094.stdout:Hit:7 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2021-02-11T09:40:24.141 INFO:teuthology.orchestra.run.smithi094.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main i386 Packages [3,096 B] 2021-02-11T09:40:24.142 INFO:teuthology.orchestra.run.smithi094.stdout:Hit:9 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2021-02-11T09:40:24.142 INFO:teuthology.orchestra.run.smithi094.stdout:Fetched 27.3 kB in 1s (38.8 kB/s) 2021-02-11T09:40:24.142 INFO:teuthology.orchestra.run.smithi094.stdout:Reading package lists... 2021-02-11T09:40:24.142 DEBUG:teuthology.orchestra.run.smithi098:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.0.0-735-g5e5bb33c-1bionic cephadm=17.0.0-735-g5e5bb33c-1bionic ceph-mds=17.0.0-735-g5e5bb33c-1bionic ceph-mgr=17.0.0-735-g5e5bb33c-1bionic ceph-common=17.0.0-735-g5e5bb33c-1bionic ceph-fuse=17.0.0-735-g5e5bb33c-1bionic ceph-test=17.0.0-735-g5e5bb33c-1bionic radosgw=17.0.0-735-g5e5bb33c-1bionic python3-rados=17.0.0-735-g5e5bb33c-1bionic python3-rgw=17.0.0-735-g5e5bb33c-1bionic python3-cephfs=17.0.0-735-g5e5bb33c-1bionic python3-rbd=17.0.0-735-g5e5bb33c-1bionic libcephfs2=17.0.0-735-g5e5bb33c-1bionic libcephfs-dev=17.0.0-735-g5e5bb33c-1bionic librados2=17.0.0-735-g5e5bb33c-1bionic librbd1=17.0.0-735-g5e5bb33c-1bionic rbd-fuse=17.0.0-735-g5e5bb33c-1bionic 2021-02-11T09:40:24.143 DEBUG:teuthology.orchestra.run.smithi094:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.0.0-735-g5e5bb33c-1bionic cephadm=17.0.0-735-g5e5bb33c-1bionic ceph-mds=17.0.0-735-g5e5bb33c-1bionic ceph-mgr=17.0.0-735-g5e5bb33c-1bionic ceph-common=17.0.0-735-g5e5bb33c-1bionic ceph-fuse=17.0.0-735-g5e5bb33c-1bionic ceph-test=17.0.0-735-g5e5bb33c-1bionic radosgw=17.0.0-735-g5e5bb33c-1bionic python3-rados=17.0.0-735-g5e5bb33c-1bionic python3-rgw=17.0.0-735-g5e5bb33c-1bionic python3-cephfs=17.0.0-735-g5e5bb33c-1bionic python3-rbd=17.0.0-735-g5e5bb33c-1bionic libcephfs2=17.0.0-735-g5e5bb33c-1bionic libcephfs-dev=17.0.0-735-g5e5bb33c-1bionic librados2=17.0.0-735-g5e5bb33c-1bionic librbd1=17.0.0-735-g5e5bb33c-1bionic rbd-fuse=17.0.0-735-g5e5bb33c-1bionic 2021-02-11T09:40:24.227 INFO:teuthology.orchestra.run.smithi098.stdout:Reading package lists... 2021-02-11T09:40:24.230 INFO:teuthology.orchestra.run.smithi094.stdout:Reading package lists... 2021-02-11T09:40:24.486 INFO:teuthology.orchestra.run.smithi098.stdout:Building dependency tree... 2021-02-11T09:40:24.486 INFO:teuthology.orchestra.run.smithi098.stdout:Reading state information... 2021-02-11T09:40:24.490 INFO:teuthology.orchestra.run.smithi094.stdout:Building dependency tree... 2021-02-11T09:40:24.490 INFO:teuthology.orchestra.run.smithi094.stdout:Reading state information... 2021-02-11T09:40:24.653 INFO:teuthology.orchestra.run.smithi098.stdout:The following additional packages will be installed: 2021-02-11T09:40:24.653 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2021-02-11T09:40:24.653 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd jq 2021-02-11T09:40:24.653 INFO:teuthology.orchestra.run.smithi098.stdout: libjaeger libjq1 libleveldb1v5 liblttng-ust0 liboath0 libonig4 librabbitmq4 2021-02-11T09:40:24.654 INFO:teuthology.orchestra.run.smithi098.stdout: libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libxslt1.1 nvme-cli 2021-02-11T09:40:24.654 INFO:teuthology.orchestra.run.smithi098.stdout: python-attr python-funcsigs python-pastedeploy-tpl python-pluggy python-py 2021-02-11T09:40:24.654 INFO:teuthology.orchestra.run.smithi098.stdout: python-pytest python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2021-02-11T09:40:24.655 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-argparse python3-ceph-common python3-cffi-backend 2021-02-11T09:40:24.655 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cherrypy3 python3-cryptography python3-cycler python3-dateutil 2021-02-11T09:40:24.655 INFO:teuthology.orchestra.run.smithi098.stdout: python3-decorator python3-fasteners python3-html5lib python3-httplib2 2021-02-11T09:40:24.655 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jinja2 python3-joblib python3-json-pointer python3-jsonpatch 2021-02-11T09:40:24.655 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2021-02-11T09:40:24.655 INFO:teuthology.orchestra.run.smithi098.stdout: python3-markupsafe python3-matplotlib python3-monotonic python3-numpy 2021-02-11T09:40:24.655 INFO:teuthology.orchestra.run.smithi098.stdout: python3-oauth2client python3-olefile python3-openssl python3-paste 2021-02-11T09:40:24.655 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pastedeploy python3-pastescript python3-pecan python3-pil 2021-02-11T09:40:24.655 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pluggy python3-prettytable python3-py python3-pyasn1 2021-02-11T09:40:24.655 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-modules python3-pyinotify python3-pyparsing python3-pytest 2021-02-11T09:40:24.655 INFO:teuthology.orchestra.run.smithi098.stdout: python3-repoze.lru python3-routes python3-rsa python3-scipy 2021-02-11T09:40:24.655 INFO:teuthology.orchestra.run.smithi098.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2021-02-11T09:40:24.655 INFO:teuthology.orchestra.run.smithi098.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2021-02-11T09:40:24.656 INFO:teuthology.orchestra.run.smithi098.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2021-02-11T09:40:24.656 INFO:teuthology.orchestra.run.smithi098.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2021-02-11T09:40:24.656 INFO:teuthology.orchestra.run.smithi098.stdout: python3-webtest python3-werkzeug socat xmlstarlet 2021-02-11T09:40:24.656 INFO:teuthology.orchestra.run.smithi098.stdout:Suggested packages: 2021-02-11T09:40:24.656 INFO:teuthology.orchestra.run.smithi098.stdout: python3-influxdb python-attr-doc python-funcsigs-doc subversion 2021-02-11T09:40:24.656 INFO:teuthology.orchestra.run.smithi098.stdout: python-pytest-xdist python-mock python-cryptography-doc 2021-02-11T09:40:24.656 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cryptography-vectors python-cycler-doc python3-genshi 2021-02-11T09:40:24.656 INFO:teuthology.orchestra.run.smithi098.stdout: python-jinja2-doc python3-crypto python3-lxml-dbg python-lxml-doc 2021-02-11T09:40:24.656 INFO:teuthology.orchestra.run.smithi098.stdout: python3-beaker python-mako-doc dvipng ffmpeg gir1.2-gtk-3.0 ghostscript 2021-02-11T09:40:24.656 INFO:teuthology.orchestra.run.smithi098.stdout: inkscape ipython3 python-matplotlib-doc python3-cairocffi python3-gi-cairo 2021-02-11T09:40:24.657 INFO:teuthology.orchestra.run.smithi098.stdout: python3-gobject python3-pyqt4 python3-sip python3-tornado 2021-02-11T09:40:24.657 INFO:teuthology.orchestra.run.smithi098.stdout: texlive-extra-utils texlive-latex-extra ttf-staypuft python-numpy-doc 2021-02-11T09:40:24.657 INFO:teuthology.orchestra.run.smithi098.stdout: python3-dev python3-numpy-dbg python-openssl-doc python3-openssl-dbg 2021-02-11T09:40:24.657 INFO:teuthology.orchestra.run.smithi098.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2021-02-11T09:40:24.657 INFO:teuthology.orchestra.run.smithi098.stdout: python-pecan-doc python-pil-doc python3-pil-dbg python-pyinotify-doc 2021-02-11T09:40:24.657 INFO:teuthology.orchestra.run.smithi098.stdout: python-pyparsing-doc python-scipy-doc python-setuptools-doc python3-dap 2021-02-11T09:40:24.657 INFO:teuthology.orchestra.run.smithi098.stdout: python-sklearn-doc python-sqlalchemy-doc python3-psycopg2 python3-pymysql 2021-02-11T09:40:24.657 INFO:teuthology.orchestra.run.smithi098.stdout: python3-fdb tix python3-tk-dbg python-waitress-doc python-webob-doc 2021-02-11T09:40:24.657 INFO:teuthology.orchestra.run.smithi098.stdout: python-webtest-doc python3-termcolor python3-watchdog python-werkzeug-doc 2021-02-11T09:40:24.659 INFO:teuthology.orchestra.run.smithi094.stdout:The following additional packages will be installed: 2021-02-11T09:40:24.659 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2021-02-11T09:40:24.659 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd jq 2021-02-11T09:40:24.659 INFO:teuthology.orchestra.run.smithi094.stdout: libjaeger libjq1 libleveldb1v5 liblttng-ust0 liboath0 libonig4 librabbitmq4 2021-02-11T09:40:24.660 INFO:teuthology.orchestra.run.smithi094.stdout: libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libxslt1.1 nvme-cli 2021-02-11T09:40:24.660 INFO:teuthology.orchestra.run.smithi094.stdout: python-attr python-funcsigs python-pastedeploy-tpl python-pluggy python-py 2021-02-11T09:40:24.660 INFO:teuthology.orchestra.run.smithi094.stdout: python-pytest python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2021-02-11T09:40:24.660 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ceph-argparse python3-ceph-common python3-cffi-backend 2021-02-11T09:40:24.660 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cherrypy3 python3-cryptography python3-cycler python3-dateutil 2021-02-11T09:40:24.660 INFO:teuthology.orchestra.run.smithi094.stdout: python3-decorator python3-fasteners python3-html5lib python3-httplib2 2021-02-11T09:40:24.660 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jinja2 python3-joblib python3-json-pointer python3-jsonpatch 2021-02-11T09:40:24.660 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2021-02-11T09:40:24.661 INFO:teuthology.orchestra.run.smithi094.stdout: python3-markupsafe python3-matplotlib python3-monotonic python3-numpy 2021-02-11T09:40:24.661 INFO:teuthology.orchestra.run.smithi094.stdout: python3-oauth2client python3-olefile python3-openssl python3-paste 2021-02-11T09:40:24.661 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pastedeploy python3-pastescript python3-pecan python3-pil 2021-02-11T09:40:24.661 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pluggy python3-prettytable python3-py python3-pyasn1 2021-02-11T09:40:24.661 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyasn1-modules python3-pyinotify python3-pyparsing python3-pytest 2021-02-11T09:40:24.661 INFO:teuthology.orchestra.run.smithi094.stdout: python3-repoze.lru python3-routes python3-rsa python3-scipy 2021-02-11T09:40:24.661 INFO:teuthology.orchestra.run.smithi094.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2021-02-11T09:40:24.661 INFO:teuthology.orchestra.run.smithi094.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2021-02-11T09:40:24.661 INFO:teuthology.orchestra.run.smithi094.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2021-02-11T09:40:24.661 INFO:teuthology.orchestra.run.smithi094.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2021-02-11T09:40:24.661 INFO:teuthology.orchestra.run.smithi094.stdout: python3-webtest python3-werkzeug socat xmlstarlet 2021-02-11T09:40:24.662 INFO:teuthology.orchestra.run.smithi094.stdout:Suggested packages: 2021-02-11T09:40:24.662 INFO:teuthology.orchestra.run.smithi094.stdout: python3-influxdb python-attr-doc python-funcsigs-doc subversion 2021-02-11T09:40:24.662 INFO:teuthology.orchestra.run.smithi094.stdout: python-pytest-xdist python-mock python-cryptography-doc 2021-02-11T09:40:24.662 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cryptography-vectors python-cycler-doc python3-genshi 2021-02-11T09:40:24.663 INFO:teuthology.orchestra.run.smithi094.stdout: python-jinja2-doc python3-crypto python3-lxml-dbg python-lxml-doc 2021-02-11T09:40:24.663 INFO:teuthology.orchestra.run.smithi094.stdout: python3-beaker python-mako-doc dvipng ffmpeg gir1.2-gtk-3.0 ghostscript 2021-02-11T09:40:24.663 INFO:teuthology.orchestra.run.smithi094.stdout: inkscape ipython3 python-matplotlib-doc python3-cairocffi python3-gi-cairo 2021-02-11T09:40:24.663 INFO:teuthology.orchestra.run.smithi094.stdout: python3-gobject python3-pyqt4 python3-sip python3-tornado 2021-02-11T09:40:24.663 INFO:teuthology.orchestra.run.smithi094.stdout: texlive-extra-utils texlive-latex-extra ttf-staypuft python-numpy-doc 2021-02-11T09:40:24.663 INFO:teuthology.orchestra.run.smithi094.stdout: python3-dev python3-numpy-dbg python-openssl-doc python3-openssl-dbg 2021-02-11T09:40:24.663 INFO:teuthology.orchestra.run.smithi094.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2021-02-11T09:40:24.663 INFO:teuthology.orchestra.run.smithi094.stdout: python-pecan-doc python-pil-doc python3-pil-dbg python-pyinotify-doc 2021-02-11T09:40:24.663 INFO:teuthology.orchestra.run.smithi094.stdout: python-pyparsing-doc python-scipy-doc python-setuptools-doc python3-dap 2021-02-11T09:40:24.663 INFO:teuthology.orchestra.run.smithi094.stdout: python-sklearn-doc python-sqlalchemy-doc python3-psycopg2 python3-pymysql 2021-02-11T09:40:24.663 INFO:teuthology.orchestra.run.smithi094.stdout: python3-fdb tix python3-tk-dbg python-waitress-doc python-webob-doc 2021-02-11T09:40:24.663 INFO:teuthology.orchestra.run.smithi094.stdout: python-webtest-doc python3-termcolor python3-watchdog python-werkzeug-doc 2021-02-11T09:40:24.756 INFO:teuthology.orchestra.run.smithi098.stdout:The following NEW packages will be installed: 2021-02-11T09:40:24.756 INFO:teuthology.orchestra.run.smithi098.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2021-02-11T09:40:24.756 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2021-02-11T09:40:24.756 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd ceph-test cephadm jq 2021-02-11T09:40:24.757 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs-dev libcephfs2 libjaeger libjq1 libleveldb1v5 liblttng-ust0 2021-02-11T09:40:24.757 INFO:teuthology.orchestra.run.smithi098.stdout: liboath0 libonig4 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2021-02-11T09:40:24.758 INFO:teuthology.orchestra.run.smithi098.stdout: libsnappy1v5 libxslt1.1 nvme-cli python-attr python-funcsigs 2021-02-11T09:40:24.758 INFO:teuthology.orchestra.run.smithi098.stdout: python-pastedeploy-tpl python-pluggy python-py python-pytest 2021-02-11T09:40:24.758 INFO:teuthology.orchestra.run.smithi098.stdout: python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2021-02-11T09:40:24.758 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs 2021-02-11T09:40:24.758 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cffi-backend python3-cherrypy3 python3-cryptography python3-cycler 2021-02-11T09:40:24.758 INFO:teuthology.orchestra.run.smithi098.stdout: python3-dateutil python3-decorator python3-fasteners python3-html5lib 2021-02-11T09:40:24.758 INFO:teuthology.orchestra.run.smithi098.stdout: python3-httplib2 python3-jinja2 python3-joblib python3-json-pointer 2021-02-11T09:40:24.758 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpatch python3-jwt python3-kubernetes python3-logutils 2021-02-11T09:40:24.758 INFO:teuthology.orchestra.run.smithi098.stdout: python3-lxml python3-mako python3-markupsafe python3-matplotlib 2021-02-11T09:40:24.758 INFO:teuthology.orchestra.run.smithi098.stdout: python3-monotonic python3-numpy python3-oauth2client python3-olefile 2021-02-11T09:40:24.758 INFO:teuthology.orchestra.run.smithi098.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2021-02-11T09:40:24.759 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py 2021-02-11T09:40:24.759 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyparsing 2021-02-11T09:40:24.759 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-rgw 2021-02-11T09:40:24.759 INFO:teuthology.orchestra.run.smithi098.stdout: python3-routes python3-rsa python3-scipy python3-setuptools 2021-02-11T09:40:24.759 INFO:teuthology.orchestra.run.smithi098.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2021-02-11T09:40:24.759 INFO:teuthology.orchestra.run.smithi098.stdout: python3-sklearn python3-sklearn-lib python3-sqlalchemy python3-tempita 2021-02-11T09:40:24.759 INFO:teuthology.orchestra.run.smithi098.stdout: python3-tk python3-tz python3-uritemplate python3-waitress 2021-02-11T09:40:24.759 INFO:teuthology.orchestra.run.smithi098.stdout: python3-webencodings python3-webob python3-websocket python3-webtest 2021-02-11T09:40:24.759 INFO:teuthology.orchestra.run.smithi098.stdout: python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2021-02-11T09:40:24.760 INFO:teuthology.orchestra.run.smithi098.stdout:The following packages will be upgraded: 2021-02-11T09:40:24.760 INFO:teuthology.orchestra.run.smithi098.stdout: librados2 librbd1 2021-02-11T09:40:24.766 INFO:teuthology.orchestra.run.smithi094.stdout:The following NEW packages will be installed: 2021-02-11T09:40:24.766 INFO:teuthology.orchestra.run.smithi094.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2021-02-11T09:40:24.766 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2021-02-11T09:40:24.767 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd ceph-test cephadm jq 2021-02-11T09:40:24.767 INFO:teuthology.orchestra.run.smithi094.stdout: libcephfs-dev libcephfs2 libjaeger libjq1 libleveldb1v5 liblttng-ust0 2021-02-11T09:40:24.767 INFO:teuthology.orchestra.run.smithi094.stdout: liboath0 libonig4 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2021-02-11T09:40:24.768 INFO:teuthology.orchestra.run.smithi094.stdout: libsnappy1v5 libxslt1.1 nvme-cli python-attr python-funcsigs 2021-02-11T09:40:24.768 INFO:teuthology.orchestra.run.smithi094.stdout: python-pastedeploy-tpl python-pluggy python-py python-pytest 2021-02-11T09:40:24.768 INFO:teuthology.orchestra.run.smithi094.stdout: python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2021-02-11T09:40:24.768 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs 2021-02-11T09:40:24.768 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cffi-backend python3-cherrypy3 python3-cryptography python3-cycler 2021-02-11T09:40:24.768 INFO:teuthology.orchestra.run.smithi094.stdout: python3-dateutil python3-decorator python3-fasteners python3-html5lib 2021-02-11T09:40:24.768 INFO:teuthology.orchestra.run.smithi094.stdout: python3-httplib2 python3-jinja2 python3-joblib python3-json-pointer 2021-02-11T09:40:24.768 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jsonpatch python3-jwt python3-kubernetes python3-logutils 2021-02-11T09:40:24.768 INFO:teuthology.orchestra.run.smithi094.stdout: python3-lxml python3-mako python3-markupsafe python3-matplotlib 2021-02-11T09:40:24.769 INFO:teuthology.orchestra.run.smithi094.stdout: python3-monotonic python3-numpy python3-oauth2client python3-olefile 2021-02-11T09:40:24.769 INFO:teuthology.orchestra.run.smithi094.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2021-02-11T09:40:24.769 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py 2021-02-11T09:40:24.769 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyparsing 2021-02-11T09:40:24.769 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-rgw 2021-02-11T09:40:24.769 INFO:teuthology.orchestra.run.smithi094.stdout: python3-routes python3-rsa python3-scipy python3-setuptools 2021-02-11T09:40:24.769 INFO:teuthology.orchestra.run.smithi094.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2021-02-11T09:40:24.769 INFO:teuthology.orchestra.run.smithi094.stdout: python3-sklearn python3-sklearn-lib python3-sqlalchemy python3-tempita 2021-02-11T09:40:24.769 INFO:teuthology.orchestra.run.smithi094.stdout: python3-tk python3-tz python3-uritemplate python3-waitress 2021-02-11T09:40:24.769 INFO:teuthology.orchestra.run.smithi094.stdout: python3-webencodings python3-webob python3-websocket python3-webtest 2021-02-11T09:40:24.769 INFO:teuthology.orchestra.run.smithi094.stdout: python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2021-02-11T09:40:24.770 INFO:teuthology.orchestra.run.smithi094.stdout:The following packages will be upgraded: 2021-02-11T09:40:24.770 INFO:teuthology.orchestra.run.smithi094.stdout: librados2 librbd1 2021-02-11T09:40:24.901 INFO:teuthology.orchestra.run.smithi098.stdout:2 upgraded, 111 newly installed, 0 to remove and 19 not upgraded. 2021-02-11T09:40:24.901 INFO:teuthology.orchestra.run.smithi098.stdout:Need to get 167 MB of archives. 2021-02-11T09:40:24.901 INFO:teuthology.orchestra.run.smithi098.stdout:After this operation, 778 MB of additional disk space will be used. 2021-02-11T09:40:24.901 INFO:teuthology.orchestra.run.smithi098.stdout:Get:1 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 libjaeger amd64 17.0.0-735-g5e5bb33c-1bionic [3,720 B] 2021-02-11T09:40:24.907 INFO:teuthology.orchestra.run.smithi094.stdout:2 upgraded, 111 newly installed, 0 to remove and 19 not upgraded. 2021-02-11T09:40:24.907 INFO:teuthology.orchestra.run.smithi094.stdout:Need to get 167 MB of archives. 2021-02-11T09:40:24.907 INFO:teuthology.orchestra.run.smithi094.stdout:After this operation, 778 MB of additional disk space will be used. 2021-02-11T09:40:24.907 INFO:teuthology.orchestra.run.smithi094.stdout:Get:1 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 libjaeger amd64 17.0.0-735-g5e5bb33c-1bionic [3,720 B] 2021-02-11T09:40:24.949 INFO:teuthology.orchestra.run.smithi098.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-uritemplate all 0.6-1ubuntu1 [5,126 B] 2021-02-11T09:40:24.949 INFO:teuthology.orchestra.run.smithi098.stdout:Get:3 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 librbd1 amd64 17.0.0-735-g5e5bb33c-1bionic [3,123 kB] 2021-02-11T09:40:24.953 INFO:teuthology.orchestra.run.smithi094.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 librbd1 amd64 17.0.0-735-g5e5bb33c-1bionic [3,123 kB] 2021-02-11T09:40:24.957 INFO:teuthology.orchestra.run.smithi094.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-uritemplate all 0.6-1ubuntu1 [5,126 B] 2021-02-11T09:40:25.029 INFO:teuthology.orchestra.run.smithi098.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liblttng-ust0 amd64 2.10.1-1 [154 kB] 2021-02-11T09:40:25.036 INFO:teuthology.orchestra.run.smithi094.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liblttng-ust0 amd64 2.10.1-1 [154 kB] 2021-02-11T09:40:25.173 INFO:teuthology.orchestra.run.smithi098.stdout:Get:5 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 librados2 amd64 17.0.0-735-g5e5bb33c-1bionic [3,162 kB] 2021-02-11T09:40:25.280 INFO:teuthology.orchestra.run.smithi098.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-prettytable all 0.7.2-3 [19.7 kB] 2021-02-11T09:40:25.283 INFO:teuthology.orchestra.run.smithi098.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 librabbitmq4 amd64 0.8.0-1ubuntu0.18.04.2 [33.9 kB] 2021-02-11T09:40:25.288 INFO:teuthology.orchestra.run.smithi098.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic/universe amd64 librdkafka1 amd64 0.11.3-1build1 [293 kB] 2021-02-11T09:40:25.297 INFO:teuthology.orchestra.run.smithi094.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-prettytable all 0.7.2-3 [19.7 kB] 2021-02-11T09:40:25.304 INFO:teuthology.orchestra.run.smithi094.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 librabbitmq4 amd64 0.8.0-1ubuntu0.18.04.2 [33.9 kB] 2021-02-11T09:40:25.308 INFO:teuthology.orchestra.run.smithi094.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 librados2 amd64 17.0.0-735-g5e5bb33c-1bionic [3,162 kB] 2021-02-11T09:40:25.323 INFO:teuthology.orchestra.run.smithi094.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic/universe amd64 librdkafka1 amd64 0.11.3-1build1 [293 kB] 2021-02-11T09:40:25.351 INFO:teuthology.orchestra.run.smithi098.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs2 amd64 17.0.0-735-g5e5bb33c-1bionic [665 kB] 2021-02-11T09:40:25.401 INFO:teuthology.orchestra.run.smithi098.stdout:Get:10 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 python3-rados amd64 17.0.0-735-g5e5bb33c-1bionic [340 kB] 2021-02-11T09:40:25.402 INFO:teuthology.orchestra.run.smithi098.stdout:Get:11 http://archive.ubuntu.com/ubuntu bionic/main amd64 libsnappy1v5 amd64 1.1.7-1 [16.0 kB] 2021-02-11T09:40:25.406 INFO:teuthology.orchestra.run.smithi098.stdout:Get:12 http://archive.ubuntu.com/ubuntu bionic/main amd64 libleveldb1v5 amd64 1.20-2 [136 kB] 2021-02-11T09:40:25.411 INFO:teuthology.orchestra.run.smithi098.stdout:Get:13 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-argparse all 17.0.0-735-g5e5bb33c-1bionic [21.7 kB] 2021-02-11T09:40:25.412 INFO:teuthology.orchestra.run.smithi098.stdout:Get:14 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 python3-cephfs amd64 17.0.0-735-g5e5bb33c-1bionic [163 kB] 2021-02-11T09:40:25.416 INFO:teuthology.orchestra.run.smithi094.stdout:Get:9 http://archive.ubuntu.com/ubuntu bionic/main amd64 libsnappy1v5 amd64 1.1.7-1 [16.0 kB] 2021-02-11T09:40:25.418 INFO:teuthology.orchestra.run.smithi094.stdout:Get:10 http://archive.ubuntu.com/ubuntu bionic/main amd64 libleveldb1v5 amd64 1.20-2 [136 kB] 2021-02-11T09:40:25.419 INFO:teuthology.orchestra.run.smithi098.stdout:Get:15 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-common all 17.0.0-735-g5e5bb33c-1bionic [29.2 kB] 2021-02-11T09:40:25.445 INFO:teuthology.orchestra.run.smithi098.stdout:Get:16 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 python3-rbd amd64 17.0.0-735-g5e5bb33c-1bionic [335 kB] 2021-02-11T09:40:25.446 INFO:teuthology.orchestra.run.smithi098.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liboath0 amd64 2.6.1-1 [44.7 kB] 2021-02-11T09:40:25.452 INFO:teuthology.orchestra.run.smithi098.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-dateutil all 2.6.1-1 [52.3 kB] 2021-02-11T09:40:25.458 INFO:teuthology.orchestra.run.smithi094.stdout:Get:11 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liboath0 amd64 2.6.1-1 [44.7 kB] 2021-02-11T09:40:25.459 INFO:teuthology.orchestra.run.smithi098.stdout:Get:19 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 librgw2 amd64 17.0.0-735-g5e5bb33c-1bionic [3,350 kB] 2021-02-11T09:40:25.460 INFO:teuthology.orchestra.run.smithi098.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-asn1crypto all 0.24.0-1 [72.8 kB] 2021-02-11T09:40:25.465 INFO:teuthology.orchestra.run.smithi094.stdout:Get:12 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-dateutil all 2.6.1-1 [52.3 kB] 2021-02-11T09:40:25.470 INFO:teuthology.orchestra.run.smithi094.stdout:Get:13 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-asn1crypto all 0.24.0-1 [72.8 kB] 2021-02-11T09:40:25.471 INFO:teuthology.orchestra.run.smithi098.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-cffi-backend amd64 1.11.5-1 [64.6 kB] 2021-02-11T09:40:25.480 INFO:teuthology.orchestra.run.smithi094.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-cffi-backend amd64 1.11.5-1 [64.6 kB] 2021-02-11T09:40:25.480 INFO:teuthology.orchestra.run.smithi098.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-cryptography amd64 2.1.4-1ubuntu1.4 [220 kB] 2021-02-11T09:40:25.488 INFO:teuthology.orchestra.run.smithi094.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-cryptography amd64 2.1.4-1ubuntu1.4 [220 kB] 2021-02-11T09:40:25.506 INFO:teuthology.orchestra.run.smithi098.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-openssl all 17.5.0-1ubuntu1 [41.5 kB] 2021-02-11T09:40:25.508 INFO:teuthology.orchestra.run.smithi094.stdout:Get:16 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs2 amd64 17.0.0-735-g5e5bb33c-1bionic [665 kB] 2021-02-11T09:40:25.508 INFO:teuthology.orchestra.run.smithi098.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-bcrypt amd64 3.1.4-2 [29.9 kB] 2021-02-11T09:40:25.508 INFO:teuthology.orchestra.run.smithi094.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-openssl all 17.5.0-1ubuntu1 [41.5 kB] 2021-02-11T09:40:25.517 INFO:teuthology.orchestra.run.smithi094.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-bcrypt amd64 3.1.4-2 [29.9 kB] 2021-02-11T09:40:25.522 INFO:teuthology.orchestra.run.smithi098.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cherrypy3 all 8.9.1-2 [160 kB] 2021-02-11T09:40:25.535 INFO:teuthology.orchestra.run.smithi098.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jwt all 1.5.3+ds1-1 [15.9 kB] 2021-02-11T09:40:25.536 INFO:teuthology.orchestra.run.smithi094.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cherrypy3 all 8.9.1-2 [160 kB] 2021-02-11T09:40:25.548 INFO:teuthology.orchestra.run.smithi094.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jwt all 1.5.3+ds1-1 [15.9 kB] 2021-02-11T09:40:25.564 INFO:teuthology.orchestra.run.smithi094.stdout:Get:21 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 python3-rados amd64 17.0.0-735-g5e5bb33c-1bionic [340 kB] 2021-02-11T09:40:25.577 INFO:teuthology.orchestra.run.smithi094.stdout:Get:22 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-argparse all 17.0.0-735-g5e5bb33c-1bionic [21.7 kB] 2021-02-11T09:40:25.578 INFO:teuthology.orchestra.run.smithi094.stdout:Get:23 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 python3-cephfs amd64 17.0.0-735-g5e5bb33c-1bionic [163 kB] 2021-02-11T09:40:25.601 INFO:teuthology.orchestra.run.smithi098.stdout:Get:27 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-logutils all 0.3.3-5 [16.7 kB] 2021-02-11T09:40:25.602 INFO:teuthology.orchestra.run.smithi098.stdout:Get:28 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-markupsafe amd64 1.0-1build1 [13.5 kB] 2021-02-11T09:40:25.603 INFO:teuthology.orchestra.run.smithi098.stdout:Get:29 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-mako all 1.0.7+ds1-1 [59.3 kB] 2021-02-11T09:40:25.605 INFO:teuthology.orchestra.run.smithi098.stdout:Get:30 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-simplegeneric all 0.8.1-1 [11.5 kB] 2021-02-11T09:40:25.606 INFO:teuthology.orchestra.run.smithi098.stdout:Get:31 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-singledispatch all 3.4.0.3-2 [7,022 B] 2021-02-11T09:40:25.607 INFO:teuthology.orchestra.run.smithi098.stdout:Get:32 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webob all 1:1.7.3-2fakesync1 [64.3 kB] 2021-02-11T09:40:25.610 INFO:teuthology.orchestra.run.smithi098.stdout:Get:33 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-bs4 all 4.6.0-1 [67.8 kB] 2021-02-11T09:40:25.612 INFO:teuthology.orchestra.run.smithi098.stdout:Get:34 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-waitress all 1.0.1-1 [53.4 kB] 2021-02-11T09:40:25.614 INFO:teuthology.orchestra.run.smithi094.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-logutils all 0.3.3-5 [16.7 kB] 2021-02-11T09:40:25.615 INFO:teuthology.orchestra.run.smithi094.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-markupsafe amd64 1.0-1build1 [13.5 kB] 2021-02-11T09:40:25.616 INFO:teuthology.orchestra.run.smithi094.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-mako all 1.0.7+ds1-1 [59.3 kB] 2021-02-11T09:40:25.618 INFO:teuthology.orchestra.run.smithi094.stdout:Get:27 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-simplegeneric all 0.8.1-1 [11.5 kB] 2021-02-11T09:40:25.618 INFO:teuthology.orchestra.run.smithi094.stdout:Get:28 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-singledispatch all 3.4.0.3-2 [7,022 B] 2021-02-11T09:40:25.619 INFO:teuthology.orchestra.run.smithi094.stdout:Get:29 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-common all 17.0.0-735-g5e5bb33c-1bionic [29.2 kB] 2021-02-11T09:40:25.619 INFO:teuthology.orchestra.run.smithi094.stdout:Get:30 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webob all 1:1.7.3-2fakesync1 [64.3 kB] 2021-02-11T09:40:25.620 INFO:teuthology.orchestra.run.smithi094.stdout:Get:31 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 python3-rbd amd64 17.0.0-735-g5e5bb33c-1bionic [335 kB] 2021-02-11T09:40:25.621 INFO:teuthology.orchestra.run.smithi094.stdout:Get:32 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-bs4 all 4.6.0-1 [67.8 kB] 2021-02-11T09:40:25.624 INFO:teuthology.orchestra.run.smithi094.stdout:Get:33 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-waitress all 1.0.1-1 [53.4 kB] 2021-02-11T09:40:25.631 INFO:teuthology.orchestra.run.smithi094.stdout:Get:34 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 librgw2 amd64 17.0.0-735-g5e5bb33c-1bionic [3,350 kB] 2021-02-11T09:40:25.666 INFO:teuthology.orchestra.run.smithi098.stdout:Get:35 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 python3-rgw amd64 17.0.0-735-g5e5bb33c-1bionic [99.4 kB] 2021-02-11T09:40:25.668 INFO:teuthology.orchestra.run.smithi098.stdout:Get:36 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 libradosstriper1 amd64 17.0.0-735-g5e5bb33c-1bionic [414 kB] 2021-02-11T09:40:25.682 INFO:teuthology.orchestra.run.smithi098.stdout:Get:37 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-common amd64 17.0.0-735-g5e5bb33c-1bionic [21.2 MB] 2021-02-11T09:40:25.685 INFO:teuthology.orchestra.run.smithi098.stdout:Get:38 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tempita all 0.5.2-2 [13.9 kB] 2021-02-11T09:40:25.686 INFO:teuthology.orchestra.run.smithi098.stdout:Get:39 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-paste all 2.0.3+dfsg-4ubuntu1 [456 kB] 2021-02-11T09:40:25.692 INFO:teuthology.orchestra.run.smithi094.stdout:Get:35 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tempita all 0.5.2-2 [13.9 kB] 2021-02-11T09:40:25.693 INFO:teuthology.orchestra.run.smithi094.stdout:Get:36 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-paste all 2.0.3+dfsg-4ubuntu1 [456 kB] 2021-02-11T09:40:25.764 INFO:teuthology.orchestra.run.smithi098.stdout:Get:40 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy-tpl all 1.5.2-4 [4,796 B] 2021-02-11T09:40:25.765 INFO:teuthology.orchestra.run.smithi098.stdout:Get:41 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastedeploy all 1.5.2-4 [13.4 kB] 2021-02-11T09:40:25.766 INFO:teuthology.orchestra.run.smithi098.stdout:Get:42 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webtest all 2.0.28-1ubuntu1 [27.9 kB] 2021-02-11T09:40:25.767 INFO:teuthology.orchestra.run.smithi098.stdout:Get:43 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pecan all 1.2.1-2 [86.1 kB] 2021-02-11T09:40:25.770 INFO:teuthology.orchestra.run.smithi098.stdout:Get:44 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 python3-werkzeug all 0.14.1+dfsg1-1ubuntu0.1 [174 kB] 2021-02-11T09:40:25.771 INFO:teuthology.orchestra.run.smithi094.stdout:Get:37 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy-tpl all 1.5.2-4 [4,796 B] 2021-02-11T09:40:25.771 INFO:teuthology.orchestra.run.smithi094.stdout:Get:38 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastedeploy all 1.5.2-4 [13.4 kB] 2021-02-11T09:40:25.771 INFO:teuthology.orchestra.run.smithi094.stdout:Get:39 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webtest all 2.0.28-1ubuntu1 [27.9 kB] 2021-02-11T09:40:25.772 INFO:teuthology.orchestra.run.smithi094.stdout:Get:40 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pecan all 1.2.1-2 [86.1 kB] 2021-02-11T09:40:25.776 INFO:teuthology.orchestra.run.smithi094.stdout:Get:41 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 python3-werkzeug all 0.14.1+dfsg1-1ubuntu0.1 [174 kB] 2021-02-11T09:40:25.778 INFO:teuthology.orchestra.run.smithi098.stdout:Get:45 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-jinja2 all 2.10-1ubuntu0.18.04.1 [95.4 kB] 2021-02-11T09:40:25.782 INFO:teuthology.orchestra.run.smithi098.stdout:Get:46 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-repoze.lru all 0.7-1 [11.7 kB] 2021-02-11T09:40:25.782 INFO:teuthology.orchestra.run.smithi098.stdout:Get:47 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-routes all 2.4.1-1 [28.4 kB] 2021-02-11T09:40:25.784 INFO:teuthology.orchestra.run.smithi094.stdout:Get:42 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-jinja2 all 2.10-1ubuntu0.18.04.1 [95.4 kB] 2021-02-11T09:40:25.788 INFO:teuthology.orchestra.run.smithi094.stdout:Get:43 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-repoze.lru all 0.7-1 [11.7 kB] 2021-02-11T09:40:25.789 INFO:teuthology.orchestra.run.smithi094.stdout:Get:44 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-routes all 2.4.1-1 [28.4 kB] 2021-02-11T09:40:25.844 INFO:teuthology.orchestra.run.smithi098.stdout:Get:48 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-numpy amd64 1:1.13.3-2ubuntu1 [1,943 kB] 2021-02-11T09:40:25.848 INFO:teuthology.orchestra.run.smithi094.stdout:Get:45 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-numpy amd64 1:1.13.3-2ubuntu1 [1,943 kB] 2021-02-11T09:40:25.865 INFO:teuthology.orchestra.run.smithi094.stdout:Get:46 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 python3-rgw amd64 17.0.0-735-g5e5bb33c-1bionic [99.4 kB] 2021-02-11T09:40:25.868 INFO:teuthology.orchestra.run.smithi094.stdout:Get:47 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 libradosstriper1 amd64 17.0.0-735-g5e5bb33c-1bionic [414 kB] 2021-02-11T09:40:25.882 INFO:teuthology.orchestra.run.smithi094.stdout:Get:48 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-common amd64 17.0.0-735-g5e5bb33c-1bionic [21.2 MB] 2021-02-11T09:40:25.932 INFO:teuthology.orchestra.run.smithi098.stdout:Get:49 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-decorator all 4.1.2-1 [9,364 B] 2021-02-11T09:40:25.932 INFO:teuthology.orchestra.run.smithi098.stdout:Get:50 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-scipy amd64 0.19.1-2ubuntu1 [9,619 kB] 2021-02-11T09:40:25.936 INFO:teuthology.orchestra.run.smithi094.stdout:Get:49 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-decorator all 4.1.2-1 [9,364 B] 2021-02-11T09:40:25.936 INFO:teuthology.orchestra.run.smithi094.stdout:Get:50 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-scipy amd64 0.19.1-2ubuntu1 [9,619 kB] 2021-02-11T09:40:26.214 INFO:teuthology.orchestra.run.smithi098.stdout:Get:51 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn-lib amd64 0.19.1-3 [1,429 kB] 2021-02-11T09:40:26.227 INFO:teuthology.orchestra.run.smithi094.stdout:Get:51 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn-lib amd64 0.19.1-3 [1,429 kB] 2021-02-11T09:40:26.252 INFO:teuthology.orchestra.run.smithi098.stdout:Get:52 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-joblib all 0.11-1 [109 kB] 2021-02-11T09:40:26.254 INFO:teuthology.orchestra.run.smithi094.stdout:Get:52 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-joblib all 0.11-1 [109 kB] 2021-02-11T09:40:26.254 INFO:teuthology.orchestra.run.smithi098.stdout:Get:53 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn all 0.19.1-3 [1,373 kB] 2021-02-11T09:40:26.256 INFO:teuthology.orchestra.run.smithi094.stdout:Get:53 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn all 0.19.1-3 [1,373 kB] 2021-02-11T09:40:26.293 INFO:teuthology.orchestra.run.smithi098.stdout:Get:54 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-monotonic all 1.1-2 [5,332 B] 2021-02-11T09:40:26.293 INFO:teuthology.orchestra.run.smithi098.stdout:Get:55 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-fasteners all 0.12.0-3 [13.5 kB] 2021-02-11T09:40:26.294 INFO:teuthology.orchestra.run.smithi098.stdout:Get:56 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-httplib2 all 0.9.2+dfsg-1ubuntu0.2 [28.6 kB] 2021-02-11T09:40:26.295 INFO:teuthology.orchestra.run.smithi098.stdout:Get:57 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1 all 0.4.2-3 [46.8 kB] 2021-02-11T09:40:26.296 INFO:teuthology.orchestra.run.smithi098.stdout:Get:58 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1-modules all 0.2.1-0.2 [32.9 kB] 2021-02-11T09:40:26.297 INFO:teuthology.orchestra.run.smithi098.stdout:Get:59 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-rsa all 3.4.2-1 [29.0 kB] 2021-02-11T09:40:26.298 INFO:teuthology.orchestra.run.smithi098.stdout:Get:60 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-sqlalchemy all 1.1.11+ds1-1ubuntu1 [680 kB] 2021-02-11T09:40:26.302 INFO:teuthology.orchestra.run.smithi094.stdout:Get:54 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-monotonic all 1.1-2 [5,332 B] 2021-02-11T09:40:26.302 INFO:teuthology.orchestra.run.smithi094.stdout:Get:55 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-fasteners all 0.12.0-3 [13.5 kB] 2021-02-11T09:40:26.303 INFO:teuthology.orchestra.run.smithi094.stdout:Get:56 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-httplib2 all 0.9.2+dfsg-1ubuntu0.2 [28.6 kB] 2021-02-11T09:40:26.304 INFO:teuthology.orchestra.run.smithi094.stdout:Get:57 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1 all 0.4.2-3 [46.8 kB] 2021-02-11T09:40:26.305 INFO:teuthology.orchestra.run.smithi094.stdout:Get:58 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1-modules all 0.2.1-0.2 [32.9 kB] 2021-02-11T09:40:26.305 INFO:teuthology.orchestra.run.smithi094.stdout:Get:59 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-rsa all 3.4.2-1 [29.0 kB] 2021-02-11T09:40:26.306 INFO:teuthology.orchestra.run.smithi094.stdout:Get:60 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-sqlalchemy all 1.1.11+ds1-1ubuntu1 [680 kB] 2021-02-11T09:40:26.374 INFO:teuthology.orchestra.run.smithi098.stdout:Get:61 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-oauth2client all 4.1.2-3 [61.5 kB] 2021-02-11T09:40:26.375 INFO:teuthology.orchestra.run.smithi098.stdout:Get:62 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-setuptools all 39.0.1-2 [248 kB] 2021-02-11T09:40:26.379 INFO:teuthology.orchestra.run.smithi098.stdout:Get:63 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-websocket all 0.44.0-0ubuntu2 [30.7 kB] 2021-02-11T09:40:26.379 INFO:teuthology.orchestra.run.smithi098.stdout:Get:64 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-kubernetes all 2.0.0-2ubuntu1 [181 kB] 2021-02-11T09:40:26.385 INFO:teuthology.orchestra.run.smithi094.stdout:Get:61 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-oauth2client all 4.1.2-3 [61.5 kB] 2021-02-11T09:40:26.389 INFO:teuthology.orchestra.run.smithi094.stdout:Get:62 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-setuptools all 39.0.1-2 [248 kB] 2021-02-11T09:40:26.397 INFO:teuthology.orchestra.run.smithi094.stdout:Get:63 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-websocket all 0.44.0-0ubuntu2 [30.7 kB] 2021-02-11T09:40:26.397 INFO:teuthology.orchestra.run.smithi094.stdout:Get:64 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-kubernetes all 2.0.0-2ubuntu1 [181 kB] 2021-02-11T09:40:26.400 INFO:teuthology.orchestra.run.smithi094.stdout:Get:65 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-json-pointer all 1.10-1 [7,478 B] 2021-02-11T09:40:26.400 INFO:teuthology.orchestra.run.smithi094.stdout:Get:66 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jsonpatch all 1.19+really1.16-1fakesync1 [13.4 kB] 2021-02-11T09:40:26.400 INFO:teuthology.orchestra.run.smithi094.stdout:Get:67 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libonig4 amd64 6.7.0-1 [119 kB] 2021-02-11T09:40:26.402 INFO:teuthology.orchestra.run.smithi094.stdout:Get:68 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libjq1 amd64 1.5+dfsg-2 [111 kB] 2021-02-11T09:40:26.454 INFO:teuthology.orchestra.run.smithi098.stdout:Get:65 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-json-pointer all 1.10-1 [7,478 B] 2021-02-11T09:40:26.454 INFO:teuthology.orchestra.run.smithi098.stdout:Get:66 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jsonpatch all 1.19+really1.16-1fakesync1 [13.4 kB] 2021-02-11T09:40:26.454 INFO:teuthology.orchestra.run.smithi098.stdout:Get:67 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libonig4 amd64 6.7.0-1 [119 kB] 2021-02-11T09:40:26.458 INFO:teuthology.orchestra.run.smithi098.stdout:Get:68 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libjq1 amd64 1.5+dfsg-2 [111 kB] 2021-02-11T09:40:26.461 INFO:teuthology.orchestra.run.smithi098.stdout:Get:69 http://archive.ubuntu.com/ubuntu bionic/universe amd64 jq amd64 1.5+dfsg-2 [45.6 kB] 2021-02-11T09:40:26.463 INFO:teuthology.orchestra.run.smithi094.stdout:Get:69 http://archive.ubuntu.com/ubuntu bionic/universe amd64 jq amd64 1.5+dfsg-2 [45.6 kB] 2021-02-11T09:40:26.463 INFO:teuthology.orchestra.run.smithi098.stdout:Get:70 http://archive.ubuntu.com/ubuntu bionic/main amd64 socat amd64 1.7.3.2-2ubuntu2 [342 kB] 2021-02-11T09:40:26.464 INFO:teuthology.orchestra.run.smithi094.stdout:Get:70 http://archive.ubuntu.com/ubuntu bionic/main amd64 socat amd64 1.7.3.2-2ubuntu2 [342 kB] 2021-02-11T09:40:26.533 INFO:teuthology.orchestra.run.smithi098.stdout:Get:71 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxslt1.1 amd64 1.1.29-5ubuntu0.2 [150 kB] 2021-02-11T09:40:26.539 INFO:teuthology.orchestra.run.smithi098.stdout:Get:72 http://archive.ubuntu.com/ubuntu bionic/universe amd64 xmlstarlet amd64 1.6.1-2 [382 kB] 2021-02-11T09:40:26.541 INFO:teuthology.orchestra.run.smithi094.stdout:Get:71 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxslt1.1 amd64 1.1.29-5ubuntu0.2 [150 kB] 2021-02-11T09:40:26.546 INFO:teuthology.orchestra.run.smithi098.stdout:Get:73 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 nvme-cli amd64 1.5-1ubuntu1.1 [184 kB] 2021-02-11T09:40:26.547 INFO:teuthology.orchestra.run.smithi094.stdout:Get:72 http://archive.ubuntu.com/ubuntu bionic/universe amd64 xmlstarlet amd64 1.6.1-2 [382 kB] 2021-02-11T09:40:26.549 INFO:teuthology.orchestra.run.smithi098.stdout:Get:74 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-attr all 17.4.0-2 [23.7 kB] 2021-02-11T09:40:26.555 INFO:teuthology.orchestra.run.smithi094.stdout:Get:73 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 nvme-cli amd64 1.5-1ubuntu1.1 [184 kB] 2021-02-11T09:40:26.558 INFO:teuthology.orchestra.run.smithi094.stdout:Get:74 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-attr all 17.4.0-2 [23.7 kB] 2021-02-11T09:40:26.558 INFO:teuthology.orchestra.run.smithi094.stdout:Get:75 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-funcsigs all 1.0.2-4 [13.5 kB] 2021-02-11T09:40:26.558 INFO:teuthology.orchestra.run.smithi094.stdout:Get:76 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pluggy all 0.6.0-1 [12.2 kB] 2021-02-11T09:40:26.558 INFO:teuthology.orchestra.run.smithi094.stdout:Get:77 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-py all 1.5.2-1 [65.1 kB] 2021-02-11T09:40:26.559 INFO:teuthology.orchestra.run.smithi094.stdout:Get:78 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pytest all 3.3.2-2 [142 kB] 2021-02-11T09:40:26.612 INFO:teuthology.orchestra.run.smithi098.stdout:Get:75 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-funcsigs all 1.0.2-4 [13.5 kB] 2021-02-11T09:40:26.613 INFO:teuthology.orchestra.run.smithi098.stdout:Get:76 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pluggy all 0.6.0-1 [12.2 kB] 2021-02-11T09:40:26.614 INFO:teuthology.orchestra.run.smithi098.stdout:Get:77 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-py all 1.5.2-1 [65.1 kB] 2021-02-11T09:40:26.616 INFO:teuthology.orchestra.run.smithi098.stdout:Get:78 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pytest all 3.3.2-2 [142 kB] 2021-02-11T09:40:26.619 INFO:teuthology.orchestra.run.smithi094.stdout:Get:79 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-attr all 17.4.0-2 [23.8 kB] 2021-02-11T09:40:26.620 INFO:teuthology.orchestra.run.smithi098.stdout:Get:79 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-attr all 17.4.0-2 [23.8 kB] 2021-02-11T09:40:26.621 INFO:teuthology.orchestra.run.smithi094.stdout:Get:80 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cycler all 0.10.0-1 [7,622 B] 2021-02-11T09:40:26.621 INFO:teuthology.orchestra.run.smithi098.stdout:Get:80 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cycler all 0.10.0-1 [7,622 B] 2021-02-11T09:40:26.692 INFO:teuthology.orchestra.run.smithi098.stdout:Get:81 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webencodings all 0.5-2 [10.4 kB] 2021-02-11T09:40:26.692 INFO:teuthology.orchestra.run.smithi098.stdout:Get:82 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-html5lib all 0.999999999-1 [81.9 kB] 2021-02-11T09:40:26.695 INFO:teuthology.orchestra.run.smithi098.stdout:Get:83 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-lxml amd64 4.2.1-1ubuntu0.3 [1,097 kB] 2021-02-11T09:40:26.698 INFO:teuthology.orchestra.run.smithi094.stdout:Get:81 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webencodings all 0.5-2 [10.4 kB] 2021-02-11T09:40:26.698 INFO:teuthology.orchestra.run.smithi094.stdout:Get:82 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-html5lib all 0.999999999-1 [81.9 kB] 2021-02-11T09:40:26.701 INFO:teuthology.orchestra.run.smithi094.stdout:Get:83 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-lxml amd64 4.2.1-1ubuntu0.3 [1,097 kB] 2021-02-11T09:40:26.715 INFO:teuthology.orchestra.run.smithi098.stdout:Get:84 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyparsing all 2.2.0+dfsg1-2 [52.2 kB] 2021-02-11T09:40:26.723 INFO:teuthology.orchestra.run.smithi094.stdout:Get:84 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyparsing all 2.2.0+dfsg1-2 [52.2 kB] 2021-02-11T09:40:26.724 INFO:teuthology.orchestra.run.smithi094.stdout:Get:85 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tz all 2018.3-2 [25.1 kB] 2021-02-11T09:40:26.724 INFO:teuthology.orchestra.run.smithi094.stdout:Get:86 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-matplotlib amd64 2.1.1-2ubuntu3 [3,907 kB] 2021-02-11T09:40:26.771 INFO:teuthology.orchestra.run.smithi098.stdout:Get:85 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tz all 2018.3-2 [25.1 kB] 2021-02-11T09:40:26.772 INFO:teuthology.orchestra.run.smithi098.stdout:Get:86 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-matplotlib amd64 2.1.1-2ubuntu3 [3,907 kB] 2021-02-11T09:40:26.861 INFO:teuthology.orchestra.run.smithi094.stdout:Get:87 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-olefile all 0.45.1-1 [33.3 kB] 2021-02-11T09:40:26.861 INFO:teuthology.orchestra.run.smithi094.stdout:Get:88 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastescript all 2.0.2-2 [54.6 kB] 2021-02-11T09:40:26.863 INFO:teuthology.orchestra.run.smithi094.stdout:Get:89 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-pil amd64 5.1.0-1ubuntu0.4 [329 kB] 2021-02-11T09:40:26.871 INFO:teuthology.orchestra.run.smithi094.stdout:Get:90 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pluggy all 0.6.0-1 [12.3 kB] 2021-02-11T09:40:26.871 INFO:teuthology.orchestra.run.smithi094.stdout:Get:91 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-py all 1.5.2-1 [65.2 kB] 2021-02-11T09:40:26.871 INFO:teuthology.orchestra.run.smithi098.stdout:Get:87 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-olefile all 0.45.1-1 [33.3 kB] 2021-02-11T09:40:26.872 INFO:teuthology.orchestra.run.smithi094.stdout:Get:92 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyinotify all 0.9.6-1 [24.7 kB] 2021-02-11T09:40:26.872 INFO:teuthology.orchestra.run.smithi094.stdout:Get:93 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pytest all 3.3.2-2 [142 kB] 2021-02-11T09:40:26.872 INFO:teuthology.orchestra.run.smithi098.stdout:Get:88 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastescript all 2.0.2-2 [54.6 kB] 2021-02-11T09:40:26.873 INFO:teuthology.orchestra.run.smithi098.stdout:Get:89 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-pil amd64 5.1.0-1ubuntu0.4 [329 kB] 2021-02-11T09:40:26.873 INFO:teuthology.orchestra.run.smithi094.stdout:Get:94 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-simplejson amd64 3.13.2-1 [49.6 kB] 2021-02-11T09:40:26.874 INFO:teuthology.orchestra.run.smithi094.stdout:Get:95 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-tk amd64 3.6.9-1~18.04 [119 kB] 2021-02-11T09:40:26.877 INFO:teuthology.orchestra.run.smithi098.stdout:Get:90 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pluggy all 0.6.0-1 [12.3 kB] 2021-02-11T09:40:26.877 INFO:teuthology.orchestra.run.smithi098.stdout:Get:91 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-py all 1.5.2-1 [65.2 kB] 2021-02-11T09:40:26.878 INFO:teuthology.orchestra.run.smithi098.stdout:Get:92 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyinotify all 0.9.6-1 [24.7 kB] 2021-02-11T09:40:26.878 INFO:teuthology.orchestra.run.smithi098.stdout:Get:93 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pytest all 3.3.2-2 [142 kB] 2021-02-11T09:40:26.880 INFO:teuthology.orchestra.run.smithi098.stdout:Get:94 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-simplejson amd64 3.13.2-1 [49.6 kB] 2021-02-11T09:40:26.930 INFO:teuthology.orchestra.run.smithi098.stdout:Get:95 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-tk amd64 3.6.9-1~18.04 [119 kB] 2021-02-11T09:40:27.019 INFO:teuthology.orchestra.run.smithi098.stdout:Get:96 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-base amd64 17.0.0-735-g5e5bb33c-1bionic [5,627 kB] 2021-02-11T09:40:27.365 INFO:teuthology.orchestra.run.smithi098.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-modules-core all 17.0.0-735-g5e5bb33c-1bionic [185 kB] 2021-02-11T09:40:27.372 INFO:teuthology.orchestra.run.smithi098.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr amd64 17.0.0-735-g5e5bb33c-1bionic [1,365 kB] 2021-02-11T09:40:27.376 INFO:teuthology.orchestra.run.smithi094.stdout:Get:96 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-base amd64 17.0.0-735-g5e5bb33c-1bionic [5,627 kB] 2021-02-11T09:40:27.481 INFO:teuthology.orchestra.run.smithi098.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mon amd64 17.0.0-735-g5e5bb33c-1bionic [6,661 kB] 2021-02-11T09:40:27.756 INFO:teuthology.orchestra.run.smithi094.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-modules-core all 17.0.0-735-g5e5bb33c-1bionic [185 kB] 2021-02-11T09:40:27.767 INFO:teuthology.orchestra.run.smithi094.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr amd64 17.0.0-735-g5e5bb33c-1bionic [1,365 kB] 2021-02-11T09:40:27.842 INFO:teuthology.orchestra.run.smithi094.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mon amd64 17.0.0-735-g5e5bb33c-1bionic [6,661 kB] 2021-02-11T09:40:27.875 INFO:teuthology.orchestra.run.smithi098.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-osd amd64 17.0.0-735-g5e5bb33c-1bionic [24.9 MB] 2021-02-11T09:40:28.332 INFO:teuthology.orchestra.run.smithi094.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-osd amd64 17.0.0-735-g5e5bb33c-1bionic [24.9 MB] 2021-02-11T09:40:29.087 INFO:teuthology.orchestra.run.smithi098.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 ceph amd64 17.0.0-735-g5e5bb33c-1bionic [3,808 B] 2021-02-11T09:40:29.087 INFO:teuthology.orchestra.run.smithi098.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-fuse amd64 17.0.0-735-g5e5bb33c-1bionic [770 kB] 2021-02-11T09:40:29.111 INFO:teuthology.orchestra.run.smithi098.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mds amd64 17.0.0-735-g5e5bb33c-1bionic [2,153 kB] 2021-02-11T09:40:29.181 INFO:teuthology.orchestra.run.smithi098.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 cephadm amd64 17.0.0-735-g5e5bb33c-1bionic [62.2 kB] 2021-02-11T09:40:29.183 INFO:teuthology.orchestra.run.smithi098.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-cephadm all 17.0.0-735-g5e5bb33c-1bionic [67.2 kB] 2021-02-11T09:40:29.185 INFO:teuthology.orchestra.run.smithi098.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-dashboard all 17.0.0-735-g5e5bb33c-1bionic [2,327 kB] 2021-02-11T09:40:29.262 INFO:teuthology.orchestra.run.smithi098.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-local all 17.0.0-735-g5e5bb33c-1bionic [7,885 kB] 2021-02-11T09:40:29.521 INFO:teuthology.orchestra.run.smithi098.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-k8sevents all 17.0.0-735-g5e5bb33c-1bionic [3,924 B] 2021-02-11T09:40:29.521 INFO:teuthology.orchestra.run.smithi098.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-rook all 17.0.0-735-g5e5bb33c-1bionic [23.3 kB] 2021-02-11T09:40:29.522 INFO:teuthology.orchestra.run.smithi098.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-test amd64 17.0.0-735-g5e5bb33c-1bionic [45.3 MB] 2021-02-11T09:40:31.668 INFO:teuthology.orchestra.run.smithi098.stdout:Get:111 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs-dev amd64 17.0.0-735-g5e5bb33c-1bionic [19.8 kB] 2021-02-11T09:40:31.668 INFO:teuthology.orchestra.run.smithi098.stdout:Get:112 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 radosgw amd64 17.0.0-735-g5e5bb33c-1bionic [10.3 MB] 2021-02-11T09:40:32.076 INFO:teuthology.orchestra.run.smithi094.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 ceph amd64 17.0.0-735-g5e5bb33c-1bionic [3,808 B] 2021-02-11T09:40:32.077 INFO:teuthology.orchestra.run.smithi094.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-fuse amd64 17.0.0-735-g5e5bb33c-1bionic [770 kB] 2021-02-11T09:40:32.144 INFO:teuthology.orchestra.run.smithi094.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mds amd64 17.0.0-735-g5e5bb33c-1bionic [2,153 kB] 2021-02-11T09:40:32.230 INFO:teuthology.orchestra.run.smithi098.stdout:Get:113 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 rbd-fuse amd64 17.0.0-735-g5e5bb33c-1bionic [77.7 kB] 2021-02-11T09:40:32.264 INFO:teuthology.orchestra.run.smithi094.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 cephadm amd64 17.0.0-735-g5e5bb33c-1bionic [62.2 kB] 2021-02-11T09:40:32.266 INFO:teuthology.orchestra.run.smithi094.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-cephadm all 17.0.0-735-g5e5bb33c-1bionic [67.2 kB] 2021-02-11T09:40:32.268 INFO:teuthology.orchestra.run.smithi094.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-dashboard all 17.0.0-735-g5e5bb33c-1bionic [2,327 kB] 2021-02-11T09:40:32.346 INFO:teuthology.orchestra.run.smithi094.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-local all 17.0.0-735-g5e5bb33c-1bionic [7,885 kB] 2021-02-11T09:40:32.605 INFO:teuthology.orchestra.run.smithi094.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-k8sevents all 17.0.0-735-g5e5bb33c-1bionic [3,924 B] 2021-02-11T09:40:32.605 INFO:teuthology.orchestra.run.smithi094.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-rook all 17.0.0-735-g5e5bb33c-1bionic [23.3 kB] 2021-02-11T09:40:32.606 INFO:teuthology.orchestra.run.smithi094.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 ceph-test amd64 17.0.0-735-g5e5bb33c-1bionic [45.3 MB] 2021-02-11T09:40:32.946 INFO:teuthology.orchestra.run.smithi098.stdout:Fetched 167 MB in 7s (22.4 MB/s) 2021-02-11T09:40:33.098 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-uritemplate. 2021-02-11T09:40:33.149 INFO:teuthology.orchestra.run.smithi098.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 137611 files and directories currently installed.) 2021-02-11T09:40:33.154 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../000-python3-uritemplate_0.6-1ubuntu1_all.deb ... 2021-02-11T09:40:33.196 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-uritemplate (0.6-1ubuntu1) ... 2021-02-11T09:40:33.440 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libjaeger. 2021-02-11T09:40:33.453 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../001-libjaeger_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:40:33.515 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libjaeger (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:40:33.851 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2021-02-11T09:40:33.866 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../002-liblttng-ust0_2.10.1-1_amd64.deb ... 2021-02-11T09:40:33.951 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking liblttng-ust0:amd64 (2.10.1-1) ... 2021-02-11T09:40:34.097 INFO:teuthology.orchestra.run.smithi094.stdout:Get:111 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs-dev amd64 17.0.0-735-g5e5bb33c-1bionic [19.8 kB] 2021-02-11T09:40:34.097 INFO:teuthology.orchestra.run.smithi094.stdout:Get:112 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 radosgw amd64 17.0.0-735-g5e5bb33c-1bionic [10.3 MB] 2021-02-11T09:40:34.318 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../003-librbd1_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:40:34.430 INFO:teuthology.orchestra.run.smithi094.stdout:Get:113 https://2.chacra.ceph.com/r/ceph/wip-blurpletastic/5e5bb33c614118d912ca83c2e5f2be67b3acc04f/ubuntu/bionic/flavors/default bionic/main amd64 rbd-fuse amd64 17.0.0-735-g5e5bb33c-1bionic [77.7 kB] 2021-02-11T09:40:34.439 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking librbd1 (17.0.0-735-g5e5bb33c-1bionic) over (12.2.13-0ubuntu0.18.04.6) ... 2021-02-11T09:40:35.106 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../004-librados2_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:40:35.129 INFO:teuthology.orchestra.run.smithi094.stdout:Fetched 167 MB in 10s (17.3 MB/s) 2021-02-11T09:40:35.235 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking librados2 (17.0.0-735-g5e5bb33c-1bionic) over (12.2.13-0ubuntu0.18.04.6) ... 2021-02-11T09:40:35.275 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-uritemplate. 2021-02-11T09:40:35.325 INFO:teuthology.orchestra.run.smithi094.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 137611 files and directories currently installed.) 2021-02-11T09:40:35.329 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../000-python3-uritemplate_0.6-1ubuntu1_all.deb ... 2021-02-11T09:40:35.374 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-uritemplate (0.6-1ubuntu1) ... 2021-02-11T09:40:35.602 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package libjaeger. 2021-02-11T09:40:35.618 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../001-libjaeger_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:40:35.660 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking libjaeger (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:40:35.880 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libcephfs2. 2021-02-11T09:40:35.896 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../005-libcephfs2_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:40:35.938 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2021-02-11T09:40:35.939 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libcephfs2 (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:40:35.954 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../002-liblttng-ust0_2.10.1-1_amd64.deb ... 2021-02-11T09:40:36.030 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking liblttng-ust0:amd64 (2.10.1-1) ... 2021-02-11T09:40:36.291 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-rados. 2021-02-11T09:40:36.305 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../006-python3-rados_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:40:36.349 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-rados (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:40:36.371 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../003-librbd1_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:40:36.492 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking librbd1 (17.0.0-735-g5e5bb33c-1bionic) over (12.2.13-0ubuntu0.18.04.6) ... 2021-02-11T09:40:36.710 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-ceph-argparse. 2021-02-11T09:40:36.725 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../007-python3-ceph-argparse_17.0.0-735-g5e5bb33c-1bionic_all.deb ... 2021-02-11T09:40:36.769 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-ceph-argparse (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:40:37.013 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-cephfs. 2021-02-11T09:40:37.028 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../008-python3-cephfs_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:40:37.097 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-cephfs (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:40:37.127 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../004-librados2_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:40:37.281 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking librados2 (17.0.0-735-g5e5bb33c-1bionic) over (12.2.13-0ubuntu0.18.04.6) ... 2021-02-11T09:40:37.407 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-ceph-common. 2021-02-11T09:40:37.423 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../009-python3-ceph-common_17.0.0-735-g5e5bb33c-1bionic_all.deb ... 2021-02-11T09:40:37.466 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-ceph-common (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:40:37.926 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package libcephfs2. 2021-02-11T09:40:37.941 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../005-libcephfs2_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:40:37.985 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking libcephfs2 (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:40:38.002 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-prettytable. 2021-02-11T09:40:38.017 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../010-python3-prettytable_0.7.2-3_all.deb ... 2021-02-11T09:40:38.060 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-prettytable (0.7.2-3) ... 2021-02-11T09:40:38.313 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-rbd. 2021-02-11T09:40:38.320 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-rados. 2021-02-11T09:40:38.327 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../011-python3-rbd_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:40:38.333 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../006-python3-rados_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:40:38.371 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-rados (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:40:38.371 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-rbd (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:40:38.690 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-ceph-argparse. 2021-02-11T09:40:38.705 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../007-python3-ceph-argparse_17.0.0-735-g5e5bb33c-1bionic_all.deb ... 2021-02-11T09:40:38.749 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-ceph-argparse (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:40:38.774 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package librabbitmq4:amd64. 2021-02-11T09:40:38.789 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../012-librabbitmq4_0.8.0-1ubuntu0.18.04.2_amd64.deb ... 2021-02-11T09:40:38.833 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2021-02-11T09:40:38.993 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-cephfs. 2021-02-11T09:40:39.008 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../008-python3-cephfs_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:40:39.051 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-cephfs (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:40:39.194 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package librdkafka1:amd64. 2021-02-11T09:40:39.207 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../013-librdkafka1_0.11.3-1build1_amd64.deb ... 2021-02-11T09:40:39.244 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking librdkafka1:amd64 (0.11.3-1build1) ... 2021-02-11T09:40:39.371 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-ceph-common. 2021-02-11T09:40:39.385 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../009-python3-ceph-common_17.0.0-735-g5e5bb33c-1bionic_all.deb ... 2021-02-11T09:40:39.429 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-ceph-common (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:40:39.580 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package librgw2. 2021-02-11T09:40:39.594 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../014-librgw2_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:40:39.663 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking librgw2 (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:40:39.699 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-prettytable. 2021-02-11T09:40:39.713 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../010-python3-prettytable_0.7.2-3_all.deb ... 2021-02-11T09:40:39.749 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-prettytable (0.7.2-3) ... 2021-02-11T09:40:39.993 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-rbd. 2021-02-11T09:40:40.007 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../011-python3-rbd_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:40:40.052 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-rbd (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:40:40.266 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-rgw. 2021-02-11T09:40:40.282 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../015-python3-rgw_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:40:40.324 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-rgw (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:40:40.421 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package librabbitmq4:amd64. 2021-02-11T09:40:40.436 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../012-librabbitmq4_0.8.0-1ubuntu0.18.04.2_amd64.deb ... 2021-02-11T09:40:40.479 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2021-02-11T09:40:40.660 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libsnappy1v5:amd64. 2021-02-11T09:40:40.676 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../016-libsnappy1v5_1.1.7-1_amd64.deb ... 2021-02-11T09:40:40.718 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libsnappy1v5:amd64 (1.1.7-1) ... 2021-02-11T09:40:40.791 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package librdkafka1:amd64. 2021-02-11T09:40:40.807 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../013-librdkafka1_0.11.3-1build1_amd64.deb ... 2021-02-11T09:40:40.841 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking librdkafka1:amd64 (0.11.3-1build1) ... 2021-02-11T09:40:41.022 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libleveldb1v5:amd64. 2021-02-11T09:40:41.036 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../017-libleveldb1v5_1.20-2_amd64.deb ... 2021-02-11T09:40:41.080 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libleveldb1v5:amd64 (1.20-2) ... 2021-02-11T09:40:41.152 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package librgw2. 2021-02-11T09:40:41.166 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../014-librgw2_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:40:41.211 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking librgw2 (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:40:41.491 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package liboath0. 2021-02-11T09:40:41.505 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../018-liboath0_2.6.1-1_amd64.deb ... 2021-02-11T09:40:41.549 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking liboath0 (2.6.1-1) ... 2021-02-11T09:40:41.797 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-rgw. 2021-02-11T09:40:41.810 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../015-python3-rgw_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:40:41.855 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-rgw (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:40:41.894 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libradosstriper1. 2021-02-11T09:40:41.909 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../019-libradosstriper1_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:40:41.952 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libradosstriper1 (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:40:42.158 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package libsnappy1v5:amd64. 2021-02-11T09:40:42.173 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../016-libsnappy1v5_1.1.7-1_amd64.deb ... 2021-02-11T09:40:42.208 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking libsnappy1v5:amd64 (1.1.7-1) ... 2021-02-11T09:40:42.439 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-common. 2021-02-11T09:40:42.453 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../020-ceph-common_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:40:42.497 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-common (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:40:42.503 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package libleveldb1v5:amd64. 2021-02-11T09:40:42.517 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../017-libleveldb1v5_1.20-2_amd64.deb ... 2021-02-11T09:40:42.561 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking libleveldb1v5:amd64 (1.20-2) ... 2021-02-11T09:40:42.973 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package liboath0. 2021-02-11T09:40:42.986 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../018-liboath0_2.6.1-1_amd64.deb ... 2021-02-11T09:40:43.031 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking liboath0 (2.6.1-1) ... 2021-02-11T09:40:43.334 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package libradosstriper1. 2021-02-11T09:40:43.347 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../019-libradosstriper1_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:40:43.384 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking libradosstriper1 (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:40:43.804 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package ceph-common. 2021-02-11T09:40:43.819 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../020-ceph-common_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:40:43.862 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking ceph-common (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:40:44.453 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-base. 2021-02-11T09:40:44.466 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../021-ceph-base_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:40:44.520 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-base (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:40:45.483 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-dateutil. 2021-02-11T09:40:45.496 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../022-python3-dateutil_2.6.1-1_all.deb ... 2021-02-11T09:40:45.533 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-dateutil (2.6.1-1) ... 2021-02-11T09:40:45.793 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package ceph-base. 2021-02-11T09:40:45.808 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../021-ceph-base_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:40:45.861 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking ceph-base (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:40:45.869 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-asn1crypto. 2021-02-11T09:40:45.882 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../023-python3-asn1crypto_0.24.0-1_all.deb ... 2021-02-11T09:40:45.936 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-asn1crypto (0.24.0-1) ... 2021-02-11T09:40:46.214 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-cffi-backend. 2021-02-11T09:40:46.227 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../024-python3-cffi-backend_1.11.5-1_amd64.deb ... 2021-02-11T09:40:46.263 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-cffi-backend (1.11.5-1) ... 2021-02-11T09:40:46.575 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-cryptography. 2021-02-11T09:40:46.590 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../025-python3-cryptography_2.1.4-1ubuntu1.4_amd64.deb ... 2021-02-11T09:40:46.633 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-cryptography (2.1.4-1ubuntu1.4) ... 2021-02-11T09:40:46.817 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-dateutil. 2021-02-11T09:40:46.832 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../022-python3-dateutil_2.6.1-1_all.deb ... 2021-02-11T09:40:46.875 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-dateutil (2.6.1-1) ... 2021-02-11T09:40:47.020 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-openssl. 2021-02-11T09:40:47.034 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../026-python3-openssl_17.5.0-1ubuntu1_all.deb ... 2021-02-11T09:40:47.078 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-openssl (17.5.0-1ubuntu1) ... 2021-02-11T09:40:47.220 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-asn1crypto. 2021-02-11T09:40:47.233 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../023-python3-asn1crypto_0.24.0-1_all.deb ... 2021-02-11T09:40:47.270 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-asn1crypto (0.24.0-1) ... 2021-02-11T09:40:47.356 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2021-02-11T09:40:47.369 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../027-ceph-mgr-modules-core_17.0.0-735-g5e5bb33c-1bionic_all.deb ... 2021-02-11T09:40:47.405 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-mgr-modules-core (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:40:47.540 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-cffi-backend. 2021-02-11T09:40:47.554 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../024-python3-cffi-backend_1.11.5-1_amd64.deb ... 2021-02-11T09:40:47.598 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-cffi-backend (1.11.5-1) ... 2021-02-11T09:40:47.777 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-bcrypt. 2021-02-11T09:40:47.790 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../028-python3-bcrypt_3.1.4-2_amd64.deb ... 2021-02-11T09:40:47.827 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-bcrypt (3.1.4-2) ... 2021-02-11T09:40:47.918 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-cryptography. 2021-02-11T09:40:47.931 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../025-python3-cryptography_2.1.4-1ubuntu1.4_amd64.deb ... 2021-02-11T09:40:47.968 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-cryptography (2.1.4-1ubuntu1.4) ... 2021-02-11T09:40:48.222 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-cherrypy3. 2021-02-11T09:40:48.235 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../029-python3-cherrypy3_8.9.1-2_all.deb ... 2021-02-11T09:40:48.272 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-cherrypy3 (8.9.1-2) ... 2021-02-11T09:40:48.321 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-openssl. 2021-02-11T09:40:48.335 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../026-python3-openssl_17.5.0-1ubuntu1_all.deb ... 2021-02-11T09:40:48.371 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-openssl (17.5.0-1ubuntu1) ... 2021-02-11T09:40:48.650 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-jwt. 2021-02-11T09:40:48.657 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2021-02-11T09:40:48.665 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../030-python3-jwt_1.5.3+ds1-1_all.deb ... 2021-02-11T09:40:48.671 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../027-ceph-mgr-modules-core_17.0.0-735-g5e5bb33c-1bionic_all.deb ... 2021-02-11T09:40:48.716 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking ceph-mgr-modules-core (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:40:48.716 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-jwt (1.5.3+ds1-1) ... 2021-02-11T09:40:49.036 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-logutils. 2021-02-11T09:40:49.052 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../031-python3-logutils_0.3.3-5_all.deb ... 2021-02-11T09:40:49.062 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-bcrypt. 2021-02-11T09:40:49.075 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../028-python3-bcrypt_3.1.4-2_amd64.deb ... 2021-02-11T09:40:49.094 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-logutils (0.3.3-5) ... 2021-02-11T09:40:49.112 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-bcrypt (3.1.4-2) ... 2021-02-11T09:40:49.406 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-markupsafe. 2021-02-11T09:40:49.407 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-cherrypy3. 2021-02-11T09:40:49.419 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../029-python3-cherrypy3_8.9.1-2_all.deb ... 2021-02-11T09:40:49.421 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../032-python3-markupsafe_1.0-1build1_amd64.deb ... 2021-02-11T09:40:49.457 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-cherrypy3 (8.9.1-2) ... 2021-02-11T09:40:49.464 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-markupsafe (1.0-1build1) ... 2021-02-11T09:40:49.767 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-mako. 2021-02-11T09:40:50.060 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-jwt. 2021-02-11T09:40:50.060 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../030-python3-jwt_1.5.3+ds1-1_all.deb ... 2021-02-11T09:40:50.060 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-jwt (1.5.3+ds1-1) ... 2021-02-11T09:40:50.060 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../033-python3-mako_1.0.7+ds1-1_all.deb ... 2021-02-11T09:40:50.060 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-mako (1.0.7+ds1-1) ... 2021-02-11T09:40:50.147 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-logutils. 2021-02-11T09:40:50.162 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-simplegeneric. 2021-02-11T09:40:50.163 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../031-python3-logutils_0.3.3-5_all.deb ... 2021-02-11T09:40:50.175 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../034-python3-simplegeneric_0.8.1-1_all.deb ... 2021-02-11T09:40:50.205 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-logutils (0.3.3-5) ... 2021-02-11T09:40:50.220 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-simplegeneric (0.8.1-1) ... 2021-02-11T09:40:50.490 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-singledispatch. 2021-02-11T09:40:50.505 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../035-python3-singledispatch_3.4.0.3-2_all.deb ... 2021-02-11T09:40:50.550 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-markupsafe. 2021-02-11T09:40:50.563 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../032-python3-markupsafe_1.0-1build1_amd64.deb ... 2021-02-11T09:40:50.564 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-singledispatch (3.4.0.3-2) ... 2021-02-11T09:40:50.600 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-markupsafe (1.0-1build1) ... 2021-02-11T09:40:50.814 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-webob. 2021-02-11T09:40:50.830 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../036-python3-webob_1%3a1.7.3-2fakesync1_all.deb ... 2021-02-11T09:40:50.864 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-webob (1:1.7.3-2fakesync1) ... 2021-02-11T09:40:50.895 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-mako. 2021-02-11T09:40:50.909 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../033-python3-mako_1.0.7+ds1-1_all.deb ... 2021-02-11T09:40:50.953 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-mako (1.0.7+ds1-1) ... 2021-02-11T09:40:51.186 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-bs4. 2021-02-11T09:40:51.200 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../037-python3-bs4_4.6.0-1_all.deb ... 2021-02-11T09:40:51.244 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-bs4 (4.6.0-1) ... 2021-02-11T09:40:51.256 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-simplegeneric. 2021-02-11T09:40:51.270 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../034-python3-simplegeneric_0.8.1-1_all.deb ... 2021-02-11T09:40:51.314 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-simplegeneric (0.8.1-1) ... 2021-02-11T09:40:51.589 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-waitress. 2021-02-11T09:40:51.602 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../038-python3-waitress_1.0.1-1_all.deb ... 2021-02-11T09:40:51.647 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-waitress (1.0.1-1) ... 2021-02-11T09:40:51.649 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-singledispatch. 2021-02-11T09:40:51.663 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../035-python3-singledispatch_3.4.0.3-2_all.deb ... 2021-02-11T09:40:51.700 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-singledispatch (3.4.0.3-2) ... 2021-02-11T09:40:51.975 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-tempita. 2021-02-11T09:40:51.990 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../039-python3-tempita_0.5.2-2_all.deb ... 2021-02-11T09:40:51.997 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-webob. 2021-02-11T09:40:52.012 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../036-python3-webob_1%3a1.7.3-2fakesync1_all.deb ... 2021-02-11T09:40:52.034 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-tempita (0.5.2-2) ... 2021-02-11T09:40:52.055 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-webob (1:1.7.3-2fakesync1) ... 2021-02-11T09:40:52.303 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-paste. 2021-02-11T09:40:52.319 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../040-python3-paste_2.0.3+dfsg-4ubuntu1_all.deb ... 2021-02-11T09:40:52.353 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-paste (2.0.3+dfsg-4ubuntu1) ... 2021-02-11T09:40:52.367 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-bs4. 2021-02-11T09:40:52.381 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../037-python3-bs4_4.6.0-1_all.deb ... 2021-02-11T09:40:52.425 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-bs4 (4.6.0-1) ... 2021-02-11T09:40:52.681 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2021-02-11T09:40:52.694 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../041-python-pastedeploy-tpl_1.5.2-4_all.deb ... 2021-02-11T09:40:52.729 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-waitress. 2021-02-11T09:40:52.731 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python-pastedeploy-tpl (1.5.2-4) ... 2021-02-11T09:40:52.744 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../038-python3-waitress_1.0.1-1_all.deb ... 2021-02-11T09:40:52.787 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-waitress (1.0.1-1) ... 2021-02-11T09:40:53.099 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-tempita. 2021-02-11T09:40:53.111 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../039-python3-tempita_0.5.2-2_all.deb ... 2021-02-11T09:40:53.835 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-pastedeploy. 2021-02-11T09:40:53.851 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../042-python3-pastedeploy_1.5.2-4_all.deb ... 2021-02-11T09:40:53.893 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-pastedeploy (1.5.2-4) ... 2021-02-11T09:40:53.916 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-tempita (0.5.2-2) ... 2021-02-11T09:40:54.229 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-webtest. 2021-02-11T09:40:54.246 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../043-python3-webtest_2.0.28-1ubuntu1_all.deb ... 2021-02-11T09:40:54.252 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-paste. 2021-02-11T09:40:54.266 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../040-python3-paste_2.0.3+dfsg-4ubuntu1_all.deb ... 2021-02-11T09:40:54.287 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-webtest (2.0.28-1ubuntu1) ... 2021-02-11T09:40:54.319 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-paste (2.0.3+dfsg-4ubuntu1) ... 2021-02-11T09:40:54.639 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2021-02-11T09:40:54.653 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../041-python-pastedeploy-tpl_1.5.2-4_all.deb ... 2021-02-11T09:40:54.674 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-pecan. 2021-02-11T09:40:54.687 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../044-python3-pecan_1.2.1-2_all.deb ... 2021-02-11T09:40:54.698 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python-pastedeploy-tpl (1.5.2-4) ... 2021-02-11T09:40:54.757 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-pecan (1.2.1-2) ... 2021-02-11T09:40:54.993 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-pastedeploy. 2021-02-11T09:40:55.008 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../042-python3-pastedeploy_1.5.2-4_all.deb ... 2021-02-11T09:40:55.051 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-pastedeploy (1.5.2-4) ... 2021-02-11T09:40:55.129 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-werkzeug. 2021-02-11T09:40:55.142 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../045-python3-werkzeug_0.14.1+dfsg1-1ubuntu0.1_all.deb ... 2021-02-11T09:40:55.179 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-werkzeug (0.14.1+dfsg1-1ubuntu0.1) ... 2021-02-11T09:40:55.354 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-webtest. 2021-02-11T09:40:55.369 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../043-python3-webtest_2.0.28-1ubuntu1_all.deb ... 2021-02-11T09:40:55.421 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-webtest (2.0.28-1ubuntu1) ... 2021-02-11T09:40:55.582 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-mgr. 2021-02-11T09:40:55.597 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../046-ceph-mgr_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:40:55.640 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-mgr (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:40:55.749 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-pecan. 2021-02-11T09:40:55.762 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../044-python3-pecan_1.2.1-2_all.deb ... 2021-02-11T09:40:55.799 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-pecan (1.2.1-2) ... 2021-02-11T09:40:56.086 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-mon. 2021-02-11T09:40:56.099 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../047-ceph-mon_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:40:56.136 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-mon (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:40:56.155 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-werkzeug. 2021-02-11T09:40:56.168 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../045-python3-werkzeug_0.14.1+dfsg1-1ubuntu0.1_all.deb ... 2021-02-11T09:40:56.213 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-werkzeug (0.14.1+dfsg1-1ubuntu0.1) ... 2021-02-11T09:40:56.601 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package ceph-mgr. 2021-02-11T09:40:56.614 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../046-ceph-mgr_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:40:56.659 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking ceph-mgr (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:40:57.107 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-osd. 2021-02-11T09:40:57.121 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../048-ceph-osd_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:40:57.146 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package ceph-mon. 2021-02-11T09:40:57.161 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../047-ceph-mon_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:40:57.165 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-osd (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:40:57.204 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking ceph-mon (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:40:58.136 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package ceph-osd. 2021-02-11T09:40:58.149 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../048-ceph-osd_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:40:58.194 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking ceph-osd (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:40:59.205 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph. 2021-02-11T09:40:59.213 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../049-ceph_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:40:59.256 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:40:59.600 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-fuse. 2021-02-11T09:40:59.616 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../050-ceph-fuse_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:40:59.667 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-fuse (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:41:00.088 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-mds. 2021-02-11T09:41:00.102 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../051-ceph-mds_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:41:00.146 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-mds (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:41:00.201 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package ceph. 2021-02-11T09:41:00.214 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../049-ceph_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:41:00.259 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking ceph (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:41:00.596 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package ceph-fuse. 2021-02-11T09:41:00.610 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../050-ceph-fuse_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:41:00.654 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking ceph-fuse (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:41:00.674 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package cephadm. 2021-02-11T09:41:00.689 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../052-cephadm_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:41:00.732 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking cephadm (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:41:01.067 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package ceph-mds. 2021-02-11T09:41:01.081 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../051-ceph-mds_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:41:01.085 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-jinja2. 2021-02-11T09:41:01.099 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../053-python3-jinja2_2.10-1ubuntu0.18.04.1_all.deb ... 2021-02-11T09:41:01.125 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking ceph-mds (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:41:01.127 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2021-02-11T09:41:01.466 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2021-02-11T09:41:01.481 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../054-ceph-mgr-cephadm_17.0.0-735-g5e5bb33c-1bionic_all.deb ... 2021-02-11T09:41:01.524 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-mgr-cephadm (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:41:01.637 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package cephadm. 2021-02-11T09:41:01.650 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../052-cephadm_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:41:01.689 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking cephadm (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:41:01.845 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-repoze.lru. 2021-02-11T09:41:01.858 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../055-python3-repoze.lru_0.7-1_all.deb ... 2021-02-11T09:41:01.895 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-repoze.lru (0.7-1) ... 2021-02-11T09:41:02.008 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-jinja2. 2021-02-11T09:41:02.022 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../053-python3-jinja2_2.10-1ubuntu0.18.04.1_all.deb ... 2021-02-11T09:41:02.058 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2021-02-11T09:41:02.199 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-routes. 2021-02-11T09:41:02.215 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../056-python3-routes_2.4.1-1_all.deb ... 2021-02-11T09:41:02.257 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-routes (2.4.1-1) ... 2021-02-11T09:41:02.355 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2021-02-11T09:41:02.370 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../054-ceph-mgr-cephadm_17.0.0-735-g5e5bb33c-1bionic_all.deb ... 2021-02-11T09:41:02.413 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking ceph-mgr-cephadm (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:41:02.577 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2021-02-11T09:41:02.593 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../057-ceph-mgr-dashboard_17.0.0-735-g5e5bb33c-1bionic_all.deb ... 2021-02-11T09:41:02.627 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-mgr-dashboard (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:41:02.833 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-repoze.lru. 2021-02-11T09:41:02.846 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../055-python3-repoze.lru_0.7-1_all.deb ... 2021-02-11T09:41:02.891 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-repoze.lru (0.7-1) ... 2021-02-11T09:41:03.179 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-routes. 2021-02-11T09:41:03.195 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../056-python3-routes_2.4.1-1_all.deb ... 2021-02-11T09:41:03.237 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-routes (2.4.1-1) ... 2021-02-11T09:41:03.617 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-numpy. 2021-02-11T09:41:03.633 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../058-python3-numpy_1%3a1.13.3-2ubuntu1_amd64.deb ... 2021-02-11T09:41:03.675 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-numpy (1:1.13.3-2ubuntu1) ... 2021-02-11T09:41:03.823 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2021-02-11T09:41:03.839 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../057-ceph-mgr-dashboard_17.0.0-735-g5e5bb33c-1bionic_all.deb ... 2021-02-11T09:41:03.881 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking ceph-mgr-dashboard (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:41:04.298 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-decorator. 2021-02-11T09:41:04.312 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../059-python3-decorator_4.1.2-1_all.deb ... 2021-02-11T09:41:04.357 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-decorator (4.1.2-1) ... 2021-02-11T09:41:04.701 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-scipy. 2021-02-11T09:41:04.716 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../060-python3-scipy_0.19.1-2ubuntu1_amd64.deb ... 2021-02-11T09:41:04.735 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-numpy. 2021-02-11T09:41:04.752 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../058-python3-numpy_1%3a1.13.3-2ubuntu1_amd64.deb ... 2021-02-11T09:41:04.760 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-scipy (0.19.1-2ubuntu1) ... 2021-02-11T09:41:04.793 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-numpy (1:1.13.3-2ubuntu1) ... 2021-02-11T09:41:05.449 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-decorator. 2021-02-11T09:41:05.465 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../059-python3-decorator_4.1.2-1_all.deb ... 2021-02-11T09:41:05.515 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-decorator (4.1.2-1) ... 2021-02-11T09:41:05.818 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-scipy. 2021-02-11T09:41:05.832 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../060-python3-scipy_0.19.1-2ubuntu1_amd64.deb ... 2021-02-11T09:41:05.876 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-scipy (0.19.1-2ubuntu1) ... 2021-02-11T09:41:06.138 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-sklearn-lib. 2021-02-11T09:41:06.153 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../061-python3-sklearn-lib_0.19.1-3_amd64.deb ... 2021-02-11T09:41:06.196 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-sklearn-lib (0.19.1-3) ... 2021-02-11T09:41:06.670 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-joblib. 2021-02-11T09:41:06.685 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../062-python3-joblib_0.11-1_all.deb ... 2021-02-11T09:41:06.728 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-joblib (0.11-1) ... 2021-02-11T09:41:07.092 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-sklearn. 2021-02-11T09:41:07.106 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../063-python3-sklearn_0.19.1-3_all.deb ... 2021-02-11T09:41:07.150 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-sklearn (0.19.1-3) ... 2021-02-11T09:41:07.178 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-sklearn-lib. 2021-02-11T09:41:07.193 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../061-python3-sklearn-lib_0.19.1-3_amd64.deb ... 2021-02-11T09:41:07.236 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-sklearn-lib (0.19.1-3) ... 2021-02-11T09:41:07.702 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-joblib. 2021-02-11T09:41:07.710 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2021-02-11T09:41:07.717 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../062-python3-joblib_0.11-1_all.deb ... 2021-02-11T09:41:07.725 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../064-ceph-mgr-diskprediction-local_17.0.0-735-g5e5bb33c-1bionic_all.deb ... 2021-02-11T09:41:07.761 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-joblib (0.11-1) ... 2021-02-11T09:41:07.769 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-mgr-diskprediction-local (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:41:08.115 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-sklearn. 2021-02-11T09:41:08.129 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../063-python3-sklearn_0.19.1-3_all.deb ... 2021-02-11T09:41:08.173 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-sklearn (0.19.1-3) ... 2021-02-11T09:41:08.776 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2021-02-11T09:41:08.790 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../064-ceph-mgr-diskprediction-local_17.0.0-735-g5e5bb33c-1bionic_all.deb ... 2021-02-11T09:41:08.834 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking ceph-mgr-diskprediction-local (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:41:09.318 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-monotonic. 2021-02-11T09:41:09.335 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../065-python3-monotonic_1.1-2_all.deb ... 2021-02-11T09:41:09.401 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-monotonic (1.1-2) ... 2021-02-11T09:41:09.713 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-fasteners. 2021-02-11T09:41:09.727 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../066-python3-fasteners_0.12.0-3_all.deb ... 2021-02-11T09:41:09.771 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-fasteners (0.12.0-3) ... 2021-02-11T09:41:09.964 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-monotonic. 2021-02-11T09:41:09.978 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../065-python3-monotonic_1.1-2_all.deb ... 2021-02-11T09:41:10.022 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-monotonic (1.1-2) ... 2021-02-11T09:41:10.091 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-httplib2. 2021-02-11T09:41:10.106 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../067-python3-httplib2_0.9.2+dfsg-1ubuntu0.2_all.deb ... 2021-02-11T09:41:10.150 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2021-02-11T09:41:10.325 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-fasteners. 2021-02-11T09:41:10.338 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../066-python3-fasteners_0.12.0-3_all.deb ... 2021-02-11T09:41:10.383 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-fasteners (0.12.0-3) ... 2021-02-11T09:41:10.510 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-pyasn1. 2021-02-11T09:41:10.525 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../068-python3-pyasn1_0.4.2-3_all.deb ... 2021-02-11T09:41:10.577 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-pyasn1 (0.4.2-3) ... 2021-02-11T09:41:10.694 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-httplib2. 2021-02-11T09:41:10.707 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../067-python3-httplib2_0.9.2+dfsg-1ubuntu0.2_all.deb ... 2021-02-11T09:41:10.744 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2021-02-11T09:41:10.931 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-pyasn1-modules. 2021-02-11T09:41:10.947 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../069-python3-pyasn1-modules_0.2.1-0.2_all.deb ... 2021-02-11T09:41:10.989 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2) ... 2021-02-11T09:41:11.080 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-pyasn1. 2021-02-11T09:41:11.096 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../068-python3-pyasn1_0.4.2-3_all.deb ... 2021-02-11T09:41:11.138 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-pyasn1 (0.4.2-3) ... 2021-02-11T09:41:11.300 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-rsa. 2021-02-11T09:41:11.314 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../070-python3-rsa_3.4.2-1_all.deb ... 2021-02-11T09:41:11.358 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-rsa (3.4.2-1) ... 2021-02-11T09:41:11.458 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-pyasn1-modules. 2021-02-11T09:41:11.471 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../069-python3-pyasn1-modules_0.2.1-0.2_all.deb ... 2021-02-11T09:41:11.508 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2) ... 2021-02-11T09:41:11.704 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-sqlalchemy. 2021-02-11T09:41:11.718 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../071-python3-sqlalchemy_1.1.11+ds1-1ubuntu1_all.deb ... 2021-02-11T09:41:11.754 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2021-02-11T09:41:11.819 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-rsa. 2021-02-11T09:41:11.834 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../070-python3-rsa_3.4.2-1_all.deb ... 2021-02-11T09:41:11.877 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-rsa (3.4.2-1) ... 2021-02-11T09:41:12.189 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-sqlalchemy. 2021-02-11T09:41:12.191 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-oauth2client. 2021-02-11T09:41:12.203 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../071-python3-sqlalchemy_1.1.11+ds1-1ubuntu1_all.deb ... 2021-02-11T09:41:12.206 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../072-python3-oauth2client_4.1.2-3_all.deb ... 2021-02-11T09:41:12.241 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-oauth2client (4.1.2-3) ... 2021-02-11T09:41:12.247 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2021-02-11T09:41:12.612 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-setuptools. 2021-02-11T09:41:12.626 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../073-python3-setuptools_39.0.1-2_all.deb ... 2021-02-11T09:41:12.670 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-setuptools (39.0.1-2) ... 2021-02-11T09:41:12.718 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-oauth2client. 2021-02-11T09:41:12.733 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../072-python3-oauth2client_4.1.2-3_all.deb ... 2021-02-11T09:41:12.776 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-oauth2client (4.1.2-3) ... 2021-02-11T09:41:13.090 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-websocket. 2021-02-11T09:41:13.104 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../074-python3-websocket_0.44.0-0ubuntu2_all.deb ... 2021-02-11T09:41:13.112 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-setuptools. 2021-02-11T09:41:13.126 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../073-python3-setuptools_39.0.1-2_all.deb ... 2021-02-11T09:41:13.140 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-websocket (0.44.0-0ubuntu2) ... 2021-02-11T09:41:13.171 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-setuptools (39.0.1-2) ... 2021-02-11T09:41:13.452 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-kubernetes. 2021-02-11T09:41:13.466 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../075-python3-kubernetes_2.0.0-2ubuntu1_all.deb ... 2021-02-11T09:41:13.527 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-kubernetes (2.0.0-2ubuntu1) ... 2021-02-11T09:41:13.566 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-websocket. 2021-02-11T09:41:13.581 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../074-python3-websocket_0.44.0-0ubuntu2_all.deb ... 2021-02-11T09:41:13.632 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-websocket (0.44.0-0ubuntu2) ... 2021-02-11T09:41:13.931 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2021-02-11T09:41:13.945 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../076-ceph-mgr-k8sevents_17.0.0-735-g5e5bb33c-1bionic_all.deb ... 2021-02-11T09:41:13.969 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-kubernetes. 2021-02-11T09:41:13.983 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../075-python3-kubernetes_2.0.0-2ubuntu1_all.deb ... 2021-02-11T09:41:14.014 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-mgr-k8sevents (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:41:14.027 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-kubernetes (2.0.0-2ubuntu1) ... 2021-02-11T09:41:14.359 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-json-pointer. 2021-02-11T09:41:14.373 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../077-python3-json-pointer_1.10-1_all.deb ... 2021-02-11T09:41:14.417 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-json-pointer (1.10-1) ... 2021-02-11T09:41:14.464 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2021-02-11T09:41:14.478 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../076-ceph-mgr-k8sevents_17.0.0-735-g5e5bb33c-1bionic_all.deb ... 2021-02-11T09:41:14.522 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking ceph-mgr-k8sevents (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:41:14.796 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-jsonpatch. 2021-02-11T09:41:14.810 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../078-python3-jsonpatch_1.19+really1.16-1fakesync1_all.deb ... 2021-02-11T09:41:14.850 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-json-pointer. 2021-02-11T09:41:14.857 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2021-02-11T09:41:14.863 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../077-python3-json-pointer_1.10-1_all.deb ... 2021-02-11T09:41:14.908 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-json-pointer (1.10-1) ... 2021-02-11T09:41:15.193 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-mgr-rook. 2021-02-11T09:41:15.207 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../079-ceph-mgr-rook_17.0.0-735-g5e5bb33c-1bionic_all.deb ... 2021-02-11T09:41:15.251 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-mgr-rook (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:41:15.269 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-jsonpatch. 2021-02-11T09:41:15.286 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../078-python3-jsonpatch_1.19+really1.16-1fakesync1_all.deb ... 2021-02-11T09:41:15.341 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2021-02-11T09:41:15.596 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libonig4:amd64. 2021-02-11T09:41:15.612 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../080-libonig4_6.7.0-1_amd64.deb ... 2021-02-11T09:41:15.641 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package ceph-mgr-rook. 2021-02-11T09:41:15.655 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../079-ceph-mgr-rook_17.0.0-735-g5e5bb33c-1bionic_all.deb ... 2021-02-11T09:41:15.700 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking ceph-mgr-rook (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:41:15.732 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libonig4:amd64 (6.7.0-1) ... 2021-02-11T09:41:16.027 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package libonig4:amd64. 2021-02-11T09:41:16.042 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../080-libonig4_6.7.0-1_amd64.deb ... 2021-02-11T09:41:16.118 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libjq1:amd64. 2021-02-11T09:41:16.133 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../081-libjq1_1.5+dfsg-2_amd64.deb ... 2021-02-11T09:41:16.171 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking libonig4:amd64 (6.7.0-1) ... 2021-02-11T09:41:16.176 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libjq1:amd64 (1.5+dfsg-2) ... 2021-02-11T09:41:16.454 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package jq. 2021-02-11T09:41:16.468 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../082-jq_1.5+dfsg-2_amd64.deb ... 2021-02-11T09:41:16.504 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking jq (1.5+dfsg-2) ... 2021-02-11T09:41:16.532 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package libjq1:amd64. 2021-02-11T09:41:16.548 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../081-libjq1_1.5+dfsg-2_amd64.deb ... 2021-02-11T09:41:16.590 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking libjq1:amd64 (1.5+dfsg-2) ... 2021-02-11T09:41:16.799 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package socat. 2021-02-11T09:41:16.813 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../083-socat_1.7.3.2-2ubuntu2_amd64.deb ... 2021-02-11T09:41:16.857 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking socat (1.7.3.2-2ubuntu2) ... 2021-02-11T09:41:16.884 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package jq. 2021-02-11T09:41:16.899 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../082-jq_1.5+dfsg-2_amd64.deb ... 2021-02-11T09:41:16.943 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking jq (1.5+dfsg-2) ... 2021-02-11T09:41:17.195 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package socat. 2021-02-11T09:41:17.209 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../083-socat_1.7.3.2-2ubuntu2_amd64.deb ... 2021-02-11T09:41:17.245 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking socat (1.7.3.2-2ubuntu2) ... 2021-02-11T09:41:17.327 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libxslt1.1:amd64. 2021-02-11T09:41:17.342 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../084-libxslt1.1_1.1.29-5ubuntu0.2_amd64.deb ... 2021-02-11T09:41:17.385 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2021-02-11T09:41:17.632 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package libxslt1.1:amd64. 2021-02-11T09:41:17.648 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../084-libxslt1.1_1.1.29-5ubuntu0.2_amd64.deb ... 2021-02-11T09:41:17.680 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package xmlstarlet. 2021-02-11T09:41:17.690 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2021-02-11T09:41:17.695 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../085-xmlstarlet_1.6.1-2_amd64.deb ... 2021-02-11T09:41:17.738 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking xmlstarlet (1.6.1-2) ... 2021-02-11T09:41:17.968 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package xmlstarlet. 2021-02-11T09:41:17.982 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../085-xmlstarlet_1.6.1-2_amd64.deb ... 2021-02-11T09:41:18.026 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking xmlstarlet (1.6.1-2) ... 2021-02-11T09:41:18.118 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-test. 2021-02-11T09:41:18.133 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../086-ceph-test_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:41:18.176 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-test (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:41:18.388 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package ceph-test. 2021-02-11T09:41:18.404 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../086-ceph-test_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:41:18.446 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking ceph-test (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:41:21.575 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libcephfs-dev. 2021-02-11T09:41:27.568 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package libcephfs-dev. 2021-02-11T09:41:29.934 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../087-libcephfs-dev_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:41:29.934 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking libcephfs-dev (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:41:29.934 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package nvme-cli. 2021-02-11T09:41:29.934 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../088-nvme-cli_1.5-1ubuntu1.1_amd64.deb ... 2021-02-11T09:41:29.934 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking nvme-cli (1.5-1ubuntu1.1) ... 2021-02-11T09:41:29.934 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python-attr. 2021-02-11T09:41:29.934 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../089-python-attr_17.4.0-2_all.deb ... 2021-02-11T09:41:29.935 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python-attr (17.4.0-2) ... 2021-02-11T09:41:29.935 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python-funcsigs. 2021-02-11T09:41:29.935 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../090-python-funcsigs_1.0.2-4_all.deb ... 2021-02-11T09:41:29.935 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python-funcsigs (1.0.2-4) ... 2021-02-11T09:41:29.935 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python-pluggy. 2021-02-11T09:41:29.935 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../091-python-pluggy_0.6.0-1_all.deb ... 2021-02-11T09:41:29.935 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python-pluggy (0.6.0-1) ... 2021-02-11T09:41:29.935 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python-py. 2021-02-11T09:41:29.935 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../092-python-py_1.5.2-1_all.deb ... 2021-02-11T09:41:29.935 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python-py (1.5.2-1) ... 2021-02-11T09:41:29.935 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python-pytest. 2021-02-11T09:41:29.935 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../093-python-pytest_3.3.2-2_all.deb ... 2021-02-11T09:41:29.935 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python-pytest (3.3.2-2) ... 2021-02-11T09:41:29.935 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-attr. 2021-02-11T09:41:29.936 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../094-python3-attr_17.4.0-2_all.deb ... 2021-02-11T09:41:29.936 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-attr (17.4.0-2) ... 2021-02-11T09:41:29.936 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-cycler. 2021-02-11T09:41:29.936 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../095-python3-cycler_0.10.0-1_all.deb ... 2021-02-11T09:41:29.936 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-cycler (0.10.0-1) ... 2021-02-11T09:41:29.936 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-webencodings. 2021-02-11T09:41:29.936 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../096-python3-webencodings_0.5-2_all.deb ... 2021-02-11T09:41:29.936 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-webencodings (0.5-2) ... 2021-02-11T09:41:29.936 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-html5lib. 2021-02-11T09:41:29.936 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../097-python3-html5lib_0.999999999-1_all.deb ... 2021-02-11T09:41:29.936 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-html5lib (0.999999999-1) ... 2021-02-11T09:41:29.936 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-lxml:amd64. 2021-02-11T09:41:29.936 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../098-python3-lxml_4.2.1-1ubuntu0.3_amd64.deb ... 2021-02-11T09:41:29.937 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-lxml:amd64 (4.2.1-1ubuntu0.3) ... 2021-02-11T09:41:29.937 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-pyparsing. 2021-02-11T09:41:29.937 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../099-python3-pyparsing_2.2.0+dfsg1-2_all.deb ... 2021-02-11T09:41:29.937 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-pyparsing (2.2.0+dfsg1-2) ... 2021-02-11T09:41:29.937 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-tz. 2021-02-11T09:41:29.937 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../100-python3-tz_2018.3-2_all.deb ... 2021-02-11T09:41:29.937 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-tz (2018.3-2) ... 2021-02-11T09:41:29.937 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-matplotlib. 2021-02-11T09:41:29.937 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../101-python3-matplotlib_2.1.1-2ubuntu3_amd64.deb ... 2021-02-11T09:41:29.937 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-matplotlib (2.1.1-2ubuntu3) ... 2021-02-11T09:41:29.937 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../087-libcephfs-dev_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:41:29.937 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libcephfs-dev (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:41:29.937 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package nvme-cli. 2021-02-11T09:41:29.938 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../088-nvme-cli_1.5-1ubuntu1.1_amd64.deb ... 2021-02-11T09:41:29.938 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking nvme-cli (1.5-1ubuntu1.1) ... 2021-02-11T09:41:29.938 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python-attr. 2021-02-11T09:41:29.938 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../089-python-attr_17.4.0-2_all.deb ... 2021-02-11T09:41:29.938 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python-attr (17.4.0-2) ... 2021-02-11T09:41:29.938 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python-funcsigs. 2021-02-11T09:41:29.938 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../090-python-funcsigs_1.0.2-4_all.deb ... 2021-02-11T09:41:29.938 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python-funcsigs (1.0.2-4) ... 2021-02-11T09:41:29.938 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python-pluggy. 2021-02-11T09:41:29.938 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../091-python-pluggy_0.6.0-1_all.deb ... 2021-02-11T09:41:29.938 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python-pluggy (0.6.0-1) ... 2021-02-11T09:41:29.938 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python-py. 2021-02-11T09:41:29.939 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../092-python-py_1.5.2-1_all.deb ... 2021-02-11T09:41:29.939 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python-py (1.5.2-1) ... 2021-02-11T09:41:29.939 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python-pytest. 2021-02-11T09:41:29.939 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../093-python-pytest_3.3.2-2_all.deb ... 2021-02-11T09:41:29.939 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python-pytest (3.3.2-2) ... 2021-02-11T09:41:29.939 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-attr. 2021-02-11T09:41:29.939 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../094-python3-attr_17.4.0-2_all.deb ... 2021-02-11T09:41:29.939 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-attr (17.4.0-2) ... 2021-02-11T09:41:29.939 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-cycler. 2021-02-11T09:41:29.939 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../095-python3-cycler_0.10.0-1_all.deb ... 2021-02-11T09:41:29.939 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-cycler (0.10.0-1) ... 2021-02-11T09:41:29.939 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-webencodings. 2021-02-11T09:41:29.939 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../096-python3-webencodings_0.5-2_all.deb ... 2021-02-11T09:41:29.940 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-webencodings (0.5-2) ... 2021-02-11T09:41:29.940 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-html5lib. 2021-02-11T09:41:29.940 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../097-python3-html5lib_0.999999999-1_all.deb ... 2021-02-11T09:41:29.940 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-html5lib (0.999999999-1) ... 2021-02-11T09:41:29.940 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-lxml:amd64. 2021-02-11T09:41:29.940 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../098-python3-lxml_4.2.1-1ubuntu0.3_amd64.deb ... 2021-02-11T09:41:29.940 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-lxml:amd64 (4.2.1-1ubuntu0.3) ... 2021-02-11T09:41:29.940 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-pyparsing. 2021-02-11T09:41:29.940 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../099-python3-pyparsing_2.2.0+dfsg1-2_all.deb ... 2021-02-11T09:41:29.940 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-pyparsing (2.2.0+dfsg1-2) ... 2021-02-11T09:41:29.940 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-tz. 2021-02-11T09:41:29.940 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../100-python3-tz_2018.3-2_all.deb ... 2021-02-11T09:41:29.940 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-tz (2018.3-2) ... 2021-02-11T09:41:29.940 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-matplotlib. 2021-02-11T09:41:29.941 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../101-python3-matplotlib_2.1.1-2ubuntu3_amd64.deb ... 2021-02-11T09:41:29.941 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-matplotlib (2.1.1-2ubuntu3) ... 2021-02-11T09:41:29.943 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-olefile. 2021-02-11T09:41:29.943 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../102-python3-olefile_0.45.1-1_all.deb ... 2021-02-11T09:41:29.944 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-olefile (0.45.1-1) ... 2021-02-11T09:41:29.944 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-pastescript. 2021-02-11T09:41:29.944 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../103-python3-pastescript_2.0.2-2_all.deb ... 2021-02-11T09:41:29.944 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-pastescript (2.0.2-2) ... 2021-02-11T09:41:29.944 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-pil:amd64. 2021-02-11T09:41:29.944 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../104-python3-pil_5.1.0-1ubuntu0.4_amd64.deb ... 2021-02-11T09:41:29.944 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-pil:amd64 (5.1.0-1ubuntu0.4) ... 2021-02-11T09:41:29.944 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-pluggy. 2021-02-11T09:41:29.944 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../105-python3-pluggy_0.6.0-1_all.deb ... 2021-02-11T09:41:29.944 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-pluggy (0.6.0-1) ... 2021-02-11T09:41:29.944 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-py. 2021-02-11T09:41:29.944 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../106-python3-py_1.5.2-1_all.deb ... 2021-02-11T09:41:29.944 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-py (1.5.2-1) ... 2021-02-11T09:41:29.944 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-olefile. 2021-02-11T09:41:29.945 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../102-python3-olefile_0.45.1-1_all.deb ... 2021-02-11T09:41:29.945 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-olefile (0.45.1-1) ... 2021-02-11T09:41:29.945 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-pastescript. 2021-02-11T09:41:29.945 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../103-python3-pastescript_2.0.2-2_all.deb ... 2021-02-11T09:41:29.945 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-pastescript (2.0.2-2) ... 2021-02-11T09:41:29.945 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-pil:amd64. 2021-02-11T09:41:29.945 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../104-python3-pil_5.1.0-1ubuntu0.4_amd64.deb ... 2021-02-11T09:41:29.945 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-pil:amd64 (5.1.0-1ubuntu0.4) ... 2021-02-11T09:41:29.945 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-pluggy. 2021-02-11T09:41:29.945 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../105-python3-pluggy_0.6.0-1_all.deb ... 2021-02-11T09:41:29.945 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-pluggy (0.6.0-1) ... 2021-02-11T09:41:29.945 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-py. 2021-02-11T09:41:29.945 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../106-python3-py_1.5.2-1_all.deb ... 2021-02-11T09:41:29.945 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-py (1.5.2-1) ... 2021-02-11T09:41:29.971 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-pyinotify. 2021-02-11T09:41:29.985 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../107-python3-pyinotify_0.9.6-1_all.deb ... 2021-02-11T09:41:30.029 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-pyinotify (0.9.6-1) ... 2021-02-11T09:41:30.218 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-pyinotify. 2021-02-11T09:41:30.235 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../107-python3-pyinotify_0.9.6-1_all.deb ... 2021-02-11T09:41:30.276 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-pyinotify (0.9.6-1) ... 2021-02-11T09:41:30.340 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-pytest. 2021-02-11T09:41:30.354 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../108-python3-pytest_3.3.2-2_all.deb ... 2021-02-11T09:41:30.398 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-pytest (3.3.2-2) ... 2021-02-11T09:41:30.621 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-pytest. 2021-02-11T09:41:30.637 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../108-python3-pytest_3.3.2-2_all.deb ... 2021-02-11T09:41:30.679 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-pytest (3.3.2-2) ... 2021-02-11T09:41:30.752 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-simplejson. 2021-02-11T09:41:30.766 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../109-python3-simplejson_3.13.2-1_amd64.deb ... 2021-02-11T09:41:30.818 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-simplejson (3.13.2-1) ... 2021-02-11T09:41:31.058 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-simplejson. 2021-02-11T09:41:31.073 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../109-python3-simplejson_3.13.2-1_amd64.deb ... 2021-02-11T09:41:31.116 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-simplejson (3.13.2-1) ... 2021-02-11T09:41:31.146 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-tk:amd64. 2021-02-11T09:41:31.161 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../110-python3-tk_3.6.9-1~18.04_amd64.deb ... 2021-02-11T09:41:31.204 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-tk:amd64 (3.6.9-1~18.04) ... 2021-02-11T09:41:31.477 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-tk:amd64. 2021-02-11T09:41:31.493 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../110-python3-tk_3.6.9-1~18.04_amd64.deb ... 2021-02-11T09:41:31.535 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-tk:amd64 (3.6.9-1~18.04) ... 2021-02-11T09:41:31.708 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package radosgw. 2021-02-11T09:41:31.722 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../111-radosgw_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:41:31.774 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking radosgw (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:41:32.032 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package radosgw. 2021-02-11T09:41:32.046 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../111-radosgw_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:41:32.098 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking radosgw (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:41:32.828 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package rbd-fuse. 2021-02-11T09:41:32.842 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../112-rbd-fuse_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:41:32.886 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking rbd-fuse (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:41:33.152 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package rbd-fuse. 2021-02-11T09:41:33.166 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../112-rbd-fuse_17.0.0-735-g5e5bb33c-1bionic_amd64.deb ... 2021-02-11T09:41:33.184 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-logutils (0.3.3-5) ... 2021-02-11T09:41:33.235 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking rbd-fuse (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:41:33.499 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-werkzeug (0.14.1+dfsg1-1ubuntu0.1) ... 2021-02-11T09:41:33.561 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-logutils (0.3.3-5) ... 2021-02-11T09:41:33.865 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-werkzeug (0.14.1+dfsg1-1ubuntu0.1) ... 2021-02-11T09:41:33.892 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up librdkafka1:amd64 (0.11.3-1build1) ... 2021-02-11T09:41:34.018 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2021-02-11T09:41:34.367 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up librdkafka1:amd64 (0.11.3-1build1) ... 2021-02-11T09:41:34.526 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2021-02-11T09:41:34.862 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-pil:amd64 (5.1.0-1ubuntu0.4) ... 2021-02-11T09:41:35.153 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-pil:amd64 (5.1.0-1ubuntu0.4) ... 2021-02-11T09:41:35.247 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-simplegeneric (0.8.1-1) ... 2021-02-11T09:41:35.538 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-simplegeneric (0.8.1-1) ... 2021-02-11T09:41:35.539 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-webencodings (0.5-2) ... 2021-02-11T09:41:35.833 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python-attr (17.4.0-2) ... 2021-02-11T09:41:35.839 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-webencodings (0.5-2) ... 2021-02-11T09:41:36.050 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-cffi-backend (1.11.5-1) ... 2021-02-11T09:41:36.149 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python-attr (17.4.0-2) ... 2021-02-11T09:41:36.185 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-py (1.5.2-1) ... 2021-02-11T09:41:36.342 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-cffi-backend (1.11.5-1) ... 2021-02-11T09:41:36.468 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-py (1.5.2-1) ... 2021-02-11T09:41:36.520 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-waitress (1.0.1-1) ... 2021-02-11T09:41:36.836 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-waitress (1.0.1-1) ... 2021-02-11T09:41:36.895 INFO:teuthology.orchestra.run.smithi094.stdout:update-alternatives: using /usr/bin/waitress-serve-python3 to provide /usr/bin/waitress-serve (waitress-serve) in auto mode 2021-02-11T09:41:37.187 INFO:teuthology.orchestra.run.smithi098.stdout:update-alternatives: using /usr/bin/waitress-serve-python3 to provide /usr/bin/waitress-serve (waitress-serve) in auto mode 2021-02-11T09:41:37.197 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-olefile (0.45.1-1) ... 2021-02-11T09:41:37.498 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-ceph-argparse (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:41:37.506 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-olefile (0.45.1-1) ... 2021-02-11T09:41:37.816 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-tempita (0.5.2-2) ... 2021-02-11T09:41:37.841 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-ceph-argparse (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:41:38.119 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up socat (1.7.3.2-2ubuntu2) ... 2021-02-11T09:41:38.150 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-tempita (0.5.2-2) ... 2021-02-11T09:41:38.245 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-numpy (1:1.13.3-2ubuntu1) ... 2021-02-11T09:41:38.460 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up socat (1.7.3.2-2ubuntu2) ... 2021-02-11T09:41:38.586 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-numpy (1:1.13.3-2ubuntu1) ... 2021-02-11T09:41:39.190 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-webob (1:1.7.3-2fakesync1) ... 2021-02-11T09:41:39.525 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-bcrypt (3.1.4-2) ... 2021-02-11T09:41:39.801 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python-py (1.5.2-1) ... 2021-02-11T09:41:39.814 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-webob (1:1.7.3-2fakesync1) ... 2021-02-11T09:41:40.028 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-simplejson (3.13.2-1) ... 2021-02-11T09:41:40.157 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-bcrypt (3.1.4-2) ... 2021-02-11T09:41:40.363 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-json-pointer (1.10-1) ... 2021-02-11T09:41:40.442 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python-py (1.5.2-1) ... 2021-02-11T09:41:40.452 INFO:teuthology.orchestra.run.smithi094.stdout:update-alternatives: using /usr/bin/python3-jsonpointer to provide /usr/bin/jsonpointer (jsonpointer) in auto mode 2021-02-11T09:41:40.668 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-simplejson (3.13.2-1) ... 2021-02-11T09:41:40.689 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up libonig4:amd64 (6.7.0-1) ... 2021-02-11T09:41:40.815 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python-pluggy (0.6.0-1) ... 2021-02-11T09:41:41.003 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-json-pointer (1.10-1) ... 2021-02-11T09:41:41.018 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-singledispatch (3.4.0.3-2) ... 2021-02-11T09:41:41.093 INFO:teuthology.orchestra.run.smithi098.stdout:update-alternatives: using /usr/bin/python3-jsonpointer to provide /usr/bin/jsonpointer (jsonpointer) in auto mode 2021-02-11T09:41:41.302 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-pyparsing (2.2.0+dfsg1-2) ... 2021-02-11T09:41:41.355 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libonig4:amd64 (6.7.0-1) ... 2021-02-11T09:41:41.481 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python-pluggy (0.6.0-1) ... 2021-02-11T09:41:41.620 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-cycler (0.10.0-1) ... 2021-02-11T09:41:41.691 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-singledispatch (3.4.0.3-2) ... 2021-02-11T09:41:41.921 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-cherrypy3 (8.9.1-2) ... 2021-02-11T09:41:41.976 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-pyparsing (2.2.0+dfsg1-2) ... 2021-02-11T09:41:42.277 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-cycler (0.10.0-1) ... 2021-02-11T09:41:42.298 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-bs4 (4.6.0-1) ... 2021-02-11T09:41:42.570 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-cherrypy3 (8.9.1-2) ... 2021-02-11T09:41:42.633 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-pluggy (0.6.0-1) ... 2021-02-11T09:41:42.934 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up cephadm (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:41:42.946 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-bs4 (4.6.0-1) ... 2021-02-11T09:41:43.281 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-pluggy (0.6.0-1) ... 2021-02-11T09:41:43.593 INFO:teuthology.orchestra.run.smithi094.stdout:Adding system user cephadm....done 2021-02-11T09:41:43.608 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up cephadm (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:41:43.758 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-markupsafe (1.0-1build1) ... 2021-02-11T09:41:44.067 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2021-02-11T09:41:44.202 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up nvme-cli (1.5-1ubuntu1.1) ... 2021-02-11T09:41:44.266 INFO:teuthology.orchestra.run.smithi098.stdout:Adding system user cephadm....done 2021-02-11T09:41:44.353 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-asn1crypto (0.24.0-1) ... 2021-02-11T09:41:44.448 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-markupsafe (1.0-1build1) ... 2021-02-11T09:41:44.697 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-prettytable (0.7.2-3) ... 2021-02-11T09:41:44.749 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2021-02-11T09:41:44.912 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up nvme-cli (1.5-1ubuntu1.1) ... 2021-02-11T09:41:45.051 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-asn1crypto (0.24.0-1) ... 2021-02-11T09:41:45.106 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up libjq1:amd64 (1.5+dfsg-2) ... 2021-02-11T09:41:45.232 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-pyasn1 (0.4.2-3) ... 2021-02-11T09:41:45.386 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-prettytable (0.7.2-3) ... 2021-02-11T09:41:45.584 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-ceph-common (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:41:45.704 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libjq1:amd64 (1.5+dfsg-2) ... 2021-02-11T09:41:45.830 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-pyasn1 (0.4.2-3) ... 2021-02-11T09:41:45.910 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-monotonic (1.1-2) ... 2021-02-11T09:41:46.190 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-ceph-common (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:41:46.203 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up libsnappy1v5:amd64 (1.1.7-1) ... 2021-02-11T09:41:46.329 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2021-02-11T09:41:46.517 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-monotonic (1.1-2) ... 2021-02-11T09:41:46.680 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-html5lib (0.999999999-1) ... 2021-02-11T09:41:46.801 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libsnappy1v5:amd64 (1.1.7-1) ... 2021-02-11T09:41:46.910 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2021-02-11T09:41:47.032 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2021-02-11T09:41:47.122 INFO:teuthology.orchestra.run.smithi094.stdout:update-alternatives: using /usr/bin/python3-jsondiff to provide /usr/bin/jsondiff (jsondiff) in auto mode 2021-02-11T09:41:47.154 INFO:teuthology.orchestra.run.smithi094.stdout:update-alternatives: using /usr/bin/python3-jsonpatch to provide /usr/bin/jsonpatch (jsonpatch) in auto mode 2021-02-11T09:41:47.270 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-html5lib (0.999999999-1) ... 2021-02-11T09:41:47.418 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-repoze.lru (0.7-1) ... 2021-02-11T09:41:47.596 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2021-02-11T09:41:47.668 INFO:teuthology.orchestra.run.smithi098.stdout:update-alternatives: using /usr/bin/python3-jsondiff to provide /usr/bin/jsondiff (jsondiff) in auto mode 2021-02-11T09:41:47.702 INFO:teuthology.orchestra.run.smithi098.stdout:update-alternatives: using /usr/bin/python3-jsonpatch to provide /usr/bin/jsonpatch (jsonpatch) in auto mode 2021-02-11T09:41:47.719 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2021-02-11T09:41:47.940 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-repoze.lru (0.7-1) ... 2021-02-11T09:41:48.039 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2) ... 2021-02-11T09:41:48.258 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2021-02-11T09:41:48.374 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-attr (17.4.0-2) ... 2021-02-11T09:41:48.570 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2) ... 2021-02-11T09:41:48.683 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python-funcsigs (1.0.2-4) ... 2021-02-11T09:41:48.884 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up libjaeger (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:41:48.887 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-attr (17.4.0-2) ... 2021-02-11T09:41:49.027 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-paste (2.0.3+dfsg-4ubuntu1) ... 2021-02-11T09:41:49.164 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python-funcsigs (1.0.2-4) ... 2021-02-11T09:41:49.357 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libjaeger (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:41:49.462 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-websocket (0.44.0-0ubuntu2) ... 2021-02-11T09:41:49.552 INFO:teuthology.orchestra.run.smithi094.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2021-02-11T09:41:49.557 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-paste (2.0.3+dfsg-4ubuntu1) ... 2021-02-11T09:41:49.807 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up liboath0 (2.6.1-1) ... 2021-02-11T09:41:49.968 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-websocket (0.44.0-0ubuntu2) ... 2021-02-11T09:41:49.991 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2021-02-11T09:41:50.090 INFO:teuthology.orchestra.run.smithi098.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2021-02-11T09:41:50.118 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-tk:amd64 (3.6.9-1~18.04) ... 2021-02-11T09:41:50.337 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up liboath0 (2.6.1-1) ... 2021-02-11T09:41:50.361 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python-pastedeploy-tpl (1.5.2-4) ... 2021-02-11T09:41:50.488 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2021-02-11T09:41:50.512 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-setuptools (39.0.1-2) ... 2021-02-11T09:41:50.614 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-tk:amd64 (3.6.9-1~18.04) ... 2021-02-11T09:41:50.857 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python-pastedeploy-tpl (1.5.2-4) ... 2021-02-11T09:41:50.915 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up liblttng-ust0:amd64 (2.10.1-1) ... 2021-02-11T09:41:50.983 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-setuptools (39.0.1-2) ... 2021-02-11T09:41:51.016 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-decorator (4.1.2-1) ... 2021-02-11T09:41:51.276 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-tz (2018.3-2) ... 2021-02-11T09:41:51.395 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up liblttng-ust0:amd64 (2.10.1-1) ... 2021-02-11T09:41:51.495 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-decorator (4.1.2-1) ... 2021-02-11T09:41:51.561 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-uritemplate (0.6-1ubuntu1) ... 2021-02-11T09:41:51.797 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-tz (2018.3-2) ... 2021-02-11T09:41:51.862 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-jwt (1.5.3+ds1-1) ... 2021-02-11T09:41:52.099 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-uritemplate (0.6-1ubuntu1) ... 2021-02-11T09:41:52.172 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-joblib (0.11-1) ... 2021-02-11T09:41:52.400 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-jwt (1.5.3+ds1-1) ... 2021-02-11T09:41:52.532 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-routes (2.4.1-1) ... 2021-02-11T09:41:52.693 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-joblib (0.11-1) ... 2021-02-11T09:41:52.816 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-pyinotify (0.9.6-1) ... 2021-02-11T09:41:53.053 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-routes (2.4.1-1) ... 2021-02-11T09:41:53.084 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-dateutil (2.6.1-1) ... 2021-02-11T09:41:53.354 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-pyinotify (0.9.6-1) ... 2021-02-11T09:41:53.394 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-cryptography (2.1.4-1ubuntu1.4) ... 2021-02-11T09:41:53.638 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-dateutil (2.6.1-1) ... 2021-02-11T09:41:53.762 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-fasteners (0.12.0-3) ... 2021-02-11T09:41:53.932 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-cryptography (2.1.4-1ubuntu1.4) ... 2021-02-11T09:41:54.081 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up jq (1.5+dfsg-2) ... 2021-02-11T09:41:54.207 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-scipy (0.19.1-2ubuntu1) ... 2021-02-11T09:41:54.284 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-fasteners (0.12.0-3) ... 2021-02-11T09:41:54.585 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up jq (1.5+dfsg-2) ... 2021-02-11T09:41:54.711 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-scipy (0.19.1-2ubuntu1) ... 2021-02-11T09:41:55.347 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-lxml:amd64 (4.2.1-1ubuntu0.3) ... 2021-02-11T09:41:55.673 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-mako (1.0.7+ds1-1) ... 2021-02-11T09:41:55.876 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-lxml:amd64 (4.2.1-1ubuntu0.3) ... 2021-02-11T09:41:55.983 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-sklearn-lib (0.19.1-3) ... 2021-02-11T09:41:56.084 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-pytest (3.3.2-2) ... 2021-02-11T09:41:56.203 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-mako (1.0.7+ds1-1) ... 2021-02-11T09:41:56.452 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python-pytest (3.3.2-2) ... 2021-02-11T09:41:56.504 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-sklearn-lib (0.19.1-3) ... 2021-02-11T09:41:56.605 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-pytest (3.3.2-2) ... 2021-02-11T09:41:56.695 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up libleveldb1v5:amd64 (1.20-2) ... 2021-02-11T09:41:56.821 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up xmlstarlet (1.6.1-2) ... 2021-02-11T09:41:56.947 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up librados2 (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:41:56.973 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python-pytest (3.3.2-2) ... 2021-02-11T09:41:57.073 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-rsa (3.4.2-1) ... 2021-02-11T09:41:57.233 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libleveldb1v5:amd64 (1.20-2) ... 2021-02-11T09:41:57.359 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up xmlstarlet (1.6.1-2) ... 2021-02-11T09:41:57.366 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up libcephfs2 (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:41:57.467 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up libcephfs-dev (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:41:57.484 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up librados2 (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:41:57.568 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-rados (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:41:57.611 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-rsa (3.4.2-1) ... 2021-02-11T09:41:57.669 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-openssl (17.5.0-1ubuntu1) ... 2021-02-11T09:41:57.929 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libcephfs2 (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:41:57.971 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-pastedeploy (1.5.2-4) ... 2021-02-11T09:41:58.029 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libcephfs-dev (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:41:58.131 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-rados (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:41:58.264 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-oauth2client (4.1.2-3) ... 2021-02-11T09:41:58.265 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-openssl (17.5.0-1ubuntu1) ... 2021-02-11T09:41:58.566 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-pastedeploy (1.5.2-4) ... 2021-02-11T09:41:58.771 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-matplotlib (2.1.1-2ubuntu3) ... 2021-02-11T09:41:58.876 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-oauth2client (4.1.2-3) ... 2021-02-11T09:41:59.271 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-matplotlib (2.1.1-2ubuntu3) ... 2021-02-11T09:41:59.478 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-sklearn (0.19.1-3) ... 2021-02-11T09:42:00.099 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-sklearn (0.19.1-3) ... 2021-02-11T09:42:00.317 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-webtest (2.0.28-1ubuntu1) ... 2021-02-11T09:42:00.585 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up libradosstriper1 (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:00.686 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-cephfs (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:00.790 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up librgw2 (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:00.895 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-webtest (2.0.28-1ubuntu1) ... 2021-02-11T09:42:00.908 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up ceph-fuse (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:01.111 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2021-02-11T09:42:01.111 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2021-02-11T09:42:01.172 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libradosstriper1 (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:01.289 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-cephfs (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:01.376 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up librbd1 (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:01.418 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up librgw2 (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:01.477 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-pecan (1.2.1-2) ... 2021-02-11T09:42:01.536 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-fuse (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:01.551 INFO:teuthology.orchestra.run.smithi094.stdout:update-alternatives: using /usr/bin/python3-pecan to provide /usr/bin/pecan (pecan) in auto mode 2021-02-11T09:42:01.591 INFO:teuthology.orchestra.run.smithi094.stdout:update-alternatives: using /usr/bin/python3-gunicorn_pecan to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode 2021-02-11T09:42:01.739 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2021-02-11T09:42:01.739 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2021-02-11T09:42:01.898 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-kubernetes (2.0.0-2ubuntu1) ... 2021-02-11T09:42:02.030 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up librbd1 (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:02.130 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-pecan (1.2.1-2) ... 2021-02-11T09:42:02.220 INFO:teuthology.orchestra.run.smithi098.stdout:update-alternatives: using /usr/bin/python3-pecan to provide /usr/bin/pecan (pecan) in auto mode 2021-02-11T09:42:02.253 INFO:teuthology.orchestra.run.smithi098.stdout:update-alternatives: using /usr/bin/python3-gunicorn_pecan to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode 2021-02-11T09:42:02.559 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-kubernetes (2.0.0-2ubuntu1) ... 2021-02-11T09:42:02.567 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up ceph-mgr-modules-core (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:02.668 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-pastescript (2.0.2-2) ... 2021-02-11T09:42:02.969 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-rgw (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:03.079 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up rbd-fuse (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:03.205 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-rbd (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:03.228 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-mgr-modules-core (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:03.331 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up ceph-common (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:03.354 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-pastescript (2.0.2-2) ... 2021-02-11T09:42:03.648 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-rgw (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:03.757 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up rbd-fuse (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:03.807 INFO:teuthology.orchestra.run.smithi094.stdout:Adding group ceph....done 2021-02-11T09:42:03.883 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-rbd (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:04.009 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-common (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:04.177 INFO:teuthology.orchestra.run.smithi094.stdout:Adding system user ceph....done 2021-02-11T09:42:04.369 INFO:teuthology.orchestra.run.smithi094.stdout:Setting system user ceph properties....done 2021-02-11T09:42:04.523 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2021-02-11T09:42:04.526 INFO:teuthology.orchestra.run.smithi098.stdout:Adding group ceph....done 2021-02-11T09:42:04.751 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2021-02-11T09:42:04.916 INFO:teuthology.orchestra.run.smithi098.stdout:Adding system user ceph....done 2021-02-11T09:42:05.051 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up radosgw (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:05.166 INFO:teuthology.orchestra.run.smithi098.stdout:Setting system user ceph properties....done 2021-02-11T09:42:05.319 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2021-02-11T09:42:05.544 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2021-02-11T09:42:05.725 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2021-02-11T09:42:05.725 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2021-02-11T09:42:05.838 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up radosgw (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:06.099 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2021-02-11T09:42:06.099 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2021-02-11T09:42:06.144 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up ceph-test (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:06.246 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up ceph-base (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:06.499 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-test (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:06.572 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2021-02-11T09:42:06.625 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-base (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:06.839 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up ceph-mgr (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:06.965 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2021-02-11T09:42:07.059 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2021-02-11T09:42:07.060 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2021-02-11T09:42:07.235 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-mgr (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:07.387 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up ceph-mgr-diskprediction-local (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:07.457 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2021-02-11T09:42:07.457 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2021-02-11T09:42:07.534 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up ceph-osd (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:07.736 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-mgr-diskprediction-local (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:07.889 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-osd (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:07.944 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2021-02-11T09:42:07.944 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2021-02-11T09:42:08.195 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up ceph-mgr-cephadm (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:08.313 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up ceph-mgr-rook (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:08.339 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2021-02-11T09:42:08.339 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2021-02-11T09:42:08.448 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up ceph-mds (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:08.642 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-mgr-cephadm (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:08.663 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2021-02-11T09:42:08.663 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2021-02-11T09:42:08.776 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-mgr-rook (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:08.918 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-mds (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:08.949 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up ceph-mon (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:09.162 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2021-02-11T09:42:09.163 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2021-02-11T09:42:09.163 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2021-02-11T09:42:09.163 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2021-02-11T09:42:09.428 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-mon (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:09.434 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up ceph-mgr-k8sevents (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:09.535 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up ceph-mgr-dashboard (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:09.644 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up ceph (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:09.669 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2021-02-11T09:42:09.669 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2021-02-11T09:42:09.746 INFO:teuthology.orchestra.run.smithi094.stdout:Processing triggers for systemd (237-3ubuntu10.44) ... 2021-02-11T09:42:09.956 INFO:teuthology.orchestra.run.smithi094.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2021-02-11T09:42:09.962 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-mgr-k8sevents (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:10.623 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-mgr-dashboard (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:10.741 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph (17.0.0-735-g5e5bb33c-1bionic) ... 2021-02-11T09:42:10.842 INFO:teuthology.orchestra.run.smithi098.stdout:Processing triggers for systemd (237-3ubuntu10.44) ... 2021-02-11T09:42:11.044 INFO:teuthology.orchestra.run.smithi098.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2021-02-11T09:42:12.092 INFO:teuthology.orchestra.run.smithi094.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2021-02-11T09:42:12.135 INFO:teuthology.orchestra.run.smithi094.stdout:ureadahead will be reprofiled on next reboot 2021-02-11T09:42:12.184 INFO:teuthology.orchestra.run.smithi094.stdout:Processing triggers for libc-bin (2.27-3ubuntu1.4) ... 2021-02-11T09:42:12.606 INFO:teuthology.orchestra.run.smithi094.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2021-02-11T09:42:12.609 DEBUG:teuthology.parallel:result is None 2021-02-11T09:42:13.221 INFO:teuthology.orchestra.run.smithi098.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2021-02-11T09:42:13.255 INFO:teuthology.orchestra.run.smithi098.stdout:ureadahead will be reprofiled on next reboot 2021-02-11T09:42:13.305 INFO:teuthology.orchestra.run.smithi098.stdout:Processing triggers for libc-bin (2.27-3ubuntu1.4) ... 2021-02-11T09:42:13.725 INFO:teuthology.orchestra.run.smithi098.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2021-02-11T09:42:13.728 DEBUG:teuthology.parallel:result is None 2021-02-11T09:42:13.729 DEBUG:teuthology.orchestra.run.smithi094:> sudo lsb_release -is 2021-02-11T09:42:13.785 INFO:teuthology.orchestra.run.smithi094.stdout:Ubuntu 2021-02-11T09:42:13.790 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-02-11T09:42:13.790 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=5e5bb33c614118d912ca83c2e5f2be67b3acc04f 2021-02-11T09:42:13.975 DEBUG:teuthology.orchestra.run.smithi094:> dpkg-query -W -f '${Version}' ceph 2021-02-11T09:42:13.998 INFO:teuthology.orchestra.run.smithi094.stdout:17.0.0-735-g5e5bb33c-1bionic 2021-02-11T09:42:13.998 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-735-g5e5bb33c-1bionic 2021-02-11T09:42:13.998 INFO:teuthology.task.install:The correct ceph version 17.0.0-735-g5e5bb33c-1bionic is installed. 2021-02-11T09:42:13.999 DEBUG:teuthology.orchestra.run.smithi098:> sudo lsb_release -is 2021-02-11T09:42:14.055 INFO:teuthology.orchestra.run.smithi098.stdout:Ubuntu 2021-02-11T09:42:14.060 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-02-11T09:42:14.060 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=5e5bb33c614118d912ca83c2e5f2be67b3acc04f 2021-02-11T09:42:14.219 DEBUG:teuthology.orchestra.run.smithi098:> dpkg-query -W -f '${Version}' ceph 2021-02-11T09:42:14.242 INFO:teuthology.orchestra.run.smithi098.stdout:17.0.0-735-g5e5bb33c-1bionic 2021-02-11T09:42:14.242 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-735-g5e5bb33c-1bionic 2021-02-11T09:42:14.242 INFO:teuthology.task.install:The correct ceph version 17.0.0-735-g5e5bb33c-1bionic is installed. 2021-02-11T09:42:14.243 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2021-02-11T09:42:14.248 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2021-02-11T09:42:14.248 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-02-11T09:42:14.262 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2021-02-11T09:42:14.262 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-02-11T09:42:14.298 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2021-02-11T09:42:14.298 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2021-02-11T09:42:14.298 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/usr/bin/daemon-helper 2021-02-11T09:42:14.317 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-02-11T09:42:14.371 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2021-02-11T09:42:14.372 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/daemon-helper 2021-02-11T09:42:14.386 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-02-11T09:42:14.441 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2021-02-11T09:42:14.441 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2021-02-11T09:42:14.441 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/usr/bin/adjust-ulimits 2021-02-11T09:42:14.456 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-02-11T09:42:14.513 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2021-02-11T09:42:14.513 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/adjust-ulimits 2021-02-11T09:42:14.528 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-02-11T09:42:14.585 INFO:teuthology.run_tasks:Running task ceph... 2021-02-11T09:42:14.676 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2021-02-11T09:42:14.676 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 777 /var/log/ceph 2021-02-11T09:42:14.679 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 777 /var/log/ceph 2021-02-11T09:42:14.693 INFO:tasks.ceph:Disabling ceph logrotate... 2021-02-11T09:42:14.693 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -f -- /etc/logrotate.d/ceph 2021-02-11T09:42:14.733 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /etc/logrotate.d/ceph 2021-02-11T09:42:14.749 INFO:tasks.ceph:Creating extra log directories... 2021-02-11T09:42:14.749 DEBUG:teuthology.orchestra.run.smithi094:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2021-02-11T09:42:14.789 DEBUG:teuthology.orchestra.run.smithi098:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2021-02-11T09:42:14.815 INFO:tasks.ceph:Creating ceph cluster c1... 2021-02-11T09:42:14.816 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'cpu_profile': set(), 'cluster': 'c1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2021-02-11T09:42:14.816 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/aemerson-2021-02-11_09:24:13-rgw-wip-blurpletastic-distro-basic-smithi/5874501', 'branch': 'wip-blurpletastic', 'description': 'rgw/multisite/{clusters frontend/beast omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind}', 'email': 'aemerson@redhat.com', 'first_in_suite': False, 'job_id': '5874501', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'aemerson-2021-02-11_09:24:13-rgw-wip-blurpletastic-distro-basic-smithi', 'nuke-on-error': True, 'os_type': 'ubuntu', 'overrides': {'admin_socket': {'branch': 'wip-blurpletastic'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '5e5bb33c614118d912ca83c2e5f2be67b3acc04f', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': '5e5bb33c614118d912ca83c2e5f2be67b3acc04f'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}, {'endpoints': ['c2.client.1'], 'is_pubsub': True, 'name': 'test-zone4'}]}]}, 'workunit': {'branch': 'wip-blurpletastic', 'sha1': '5e5bb33c614118d912ca83c2e5f2be67b3acc04f'}}, 'owner': 'scheduled_aemerson@teuthology', 'priority': 70, 'repo': 'git://git.ceph.com/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'sha1': '5e5bb33c614118d912ca83c2e5f2be67b3acc04f', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'wip-blurpletastic', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_5e5bb33c614118d912ca83c2e5f2be67b3acc04f/qa', 'suite_relpath': 'qa', 'suite_repo': 'git://git.ceph.com/ceph-ci.git', 'suite_sha1': '5e5bb33c614118d912ca83c2e5f2be67b3acc04f', 'targets': {'smithi094.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDhsR+Lt7mvR59HsmPpNWxJNim8Xttba7vcp+ScC6u0Pswb6S42rmwG38WbZuU/7ep6pP2x3nzFqADW1Wnw7gA0s7j0f97KmM/1gwSTmn7wLM3vmH9YflZd0u92ejdlWXqzSlZGVQs6OeTXosJev3vWCuWPWzrqKD9PcoohB43I5OcEHwXdYw6yfox2Ev1xwUHJLfFoqPhuuTXsREJrjwfc7wnu0R/x2Uq6TVNZZGZAi4ZUjD0dDShWWpBVBBLR9CY/ld75UTGV43izFmc0s3LqjcZXoTL7Dy3PyhObcx5YlnexpyOFnC9WjKvvd8YM55slmtssLnOCuSO7f2gBX4XH', 'smithi098.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFrxDgXyNFGHyNBYsLuHdnF6Guzsz8LugH6RDGsPAxVb7lwHj53vn1Bo+4ZTv2cbfaDSCx15SSePh5B5Bxej/1IbcypypuZmr928LW50imJaz1UqRratL3KhgE8fC6+dkFKflLidlrAGSq3WPgRGsNq1Y+wMnGK4NC9+LUHRDlGLfPOBTZcYo/SOVg/dVnubnyp+T043bk/jvhjtHNprVq1WMV0i/CyvW46Sc9NaUetAF24NIx3lxL0jxLEcDqVV4DpcpyXK2zWpNwtNxUq1erqDESFP5p/m19cA6YiJsRqwyyYjZVWdb0ZpBTK9RZAB2iO+92elpGyJchAJ4jHaSN'}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '5e5bb33c614118d912ca83c2e5f2be67b3acc04f', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2'}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology_branch': 'master', 'timestamp': '2021-02-11_09:24:13', 'tube': 'smithi', 'user': 'aemerson', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.29888'} 2021-02-11T09:42:14.816 DEBUG:teuthology.orchestra.run.smithi094:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2021-02-11T09:42:14.845 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2021-02-11T09:42:14.853 DEBUG:teuthology.orchestra.run.smithi094:> sudo install -d -m0777 -- /var/run/ceph 2021-02-11T09:42:14.897 DEBUG:teuthology.orchestra.run.smithi098:> sudo install -d -m0777 -- /var/run/ceph 2021-02-11T09:42:14.911 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2021-02-11T09:42:14.911 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/scratch_devs of=/dev/stdout 2021-02-11T09:42:14.957 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-02-11T09:42:14.957 DEBUG:teuthology.orchestra.run.smithi094:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-02-11T09:42:15.006 INFO:teuthology.orchestra.run.smithi094.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2021-02-11T09:42:15.006 INFO:teuthology.orchestra.run.smithi094.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-11T09:42:15.006 INFO:teuthology.orchestra.run.smithi094.stdout:Device: 6h/6d Inode: 339 Links: 1 2021-02-11T09:42:15.006 INFO:teuthology.orchestra.run.smithi094.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-11T09:42:15.006 INFO:teuthology.orchestra.run.smithi094.stdout:Access: 2021-02-11 09:39:35.582868644 +0000 2021-02-11T09:42:15.006 INFO:teuthology.orchestra.run.smithi094.stdout:Modify: 2021-02-11 09:39:35.466868647 +0000 2021-02-11T09:42:15.006 INFO:teuthology.orchestra.run.smithi094.stdout:Change: 2021-02-11 09:39:35.466868647 +0000 2021-02-11T09:42:15.006 INFO:teuthology.orchestra.run.smithi094.stdout: Birth: - 2021-02-11T09:42:15.014 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records in 2021-02-11T09:42:15.015 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records out 2021-02-11T09:42:15.015 INFO:teuthology.orchestra.run.smithi094.stderr:512 bytes copied, 0.000287987 s, 1.8 MB/s 2021-02-11T09:42:15.018 DEBUG:teuthology.orchestra.run.smithi094:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-02-11T09:42:15.066 INFO:teuthology.orchestra.run.smithi094.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2021-02-11T09:42:15.066 INFO:teuthology.orchestra.run.smithi094.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-11T09:42:15.066 INFO:teuthology.orchestra.run.smithi094.stdout:Device: 6h/6d Inode: 346 Links: 1 2021-02-11T09:42:15.066 INFO:teuthology.orchestra.run.smithi094.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-11T09:42:15.066 INFO:teuthology.orchestra.run.smithi094.stdout:Access: 2021-02-11 09:39:35.790868639 +0000 2021-02-11T09:42:15.066 INFO:teuthology.orchestra.run.smithi094.stdout:Modify: 2021-02-11 09:39:35.690868642 +0000 2021-02-11T09:42:15.066 INFO:teuthology.orchestra.run.smithi094.stdout:Change: 2021-02-11 09:39:35.690868642 +0000 2021-02-11T09:42:15.066 INFO:teuthology.orchestra.run.smithi094.stdout: Birth: - 2021-02-11T09:42:15.075 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records in 2021-02-11T09:42:15.075 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records out 2021-02-11T09:42:15.075 INFO:teuthology.orchestra.run.smithi094.stderr:512 bytes copied, 0.000270833 s, 1.9 MB/s 2021-02-11T09:42:15.079 DEBUG:teuthology.orchestra.run.smithi094:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-02-11T09:42:15.126 INFO:teuthology.orchestra.run.smithi094.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2021-02-11T09:42:15.126 INFO:teuthology.orchestra.run.smithi094.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-11T09:42:15.126 INFO:teuthology.orchestra.run.smithi094.stdout:Device: 6h/6d Inode: 352 Links: 1 2021-02-11T09:42:15.126 INFO:teuthology.orchestra.run.smithi094.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-11T09:42:15.126 INFO:teuthology.orchestra.run.smithi094.stdout:Access: 2021-02-11 09:39:36.014868634 +0000 2021-02-11T09:42:15.126 INFO:teuthology.orchestra.run.smithi094.stdout:Modify: 2021-02-11 09:39:35.922868636 +0000 2021-02-11T09:42:15.126 INFO:teuthology.orchestra.run.smithi094.stdout:Change: 2021-02-11 09:39:35.922868636 +0000 2021-02-11T09:42:15.126 INFO:teuthology.orchestra.run.smithi094.stdout: Birth: - 2021-02-11T09:42:15.134 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records in 2021-02-11T09:42:15.134 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records out 2021-02-11T09:42:15.134 INFO:teuthology.orchestra.run.smithi094.stderr:512 bytes copied, 0.000227114 s, 2.3 MB/s 2021-02-11T09:42:15.138 DEBUG:teuthology.orchestra.run.smithi094:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-02-11T09:42:15.185 INFO:teuthology.orchestra.run.smithi094.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2021-02-11T09:42:15.186 INFO:teuthology.orchestra.run.smithi094.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-11T09:42:15.186 INFO:teuthology.orchestra.run.smithi094.stdout:Device: 6h/6d Inode: 358 Links: 1 2021-02-11T09:42:15.186 INFO:teuthology.orchestra.run.smithi094.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-11T09:42:15.186 INFO:teuthology.orchestra.run.smithi094.stdout:Access: 2021-02-11 09:39:36.238868629 +0000 2021-02-11T09:42:15.186 INFO:teuthology.orchestra.run.smithi094.stdout:Modify: 2021-02-11 09:39:36.146868631 +0000 2021-02-11T09:42:15.186 INFO:teuthology.orchestra.run.smithi094.stdout:Change: 2021-02-11 09:39:36.146868631 +0000 2021-02-11T09:42:15.186 INFO:teuthology.orchestra.run.smithi094.stdout: Birth: - 2021-02-11T09:42:15.195 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records in 2021-02-11T09:42:15.195 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records out 2021-02-11T09:42:15.195 INFO:teuthology.orchestra.run.smithi094.stderr:512 bytes copied, 0.00027821 s, 1.8 MB/s 2021-02-11T09:42:15.200 INFO:tasks.ceph:osd dev map: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2021-02-11T09:42:15.200 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi094.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2021-02-11T09:42:15.200 INFO:tasks.ceph:Generating config... 2021-02-11T09:42:15.202 INFO:tasks.ceph:[client] debug rgw = 20 2021-02-11T09:42:15.202 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2021-02-11T09:42:15.202 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2021-02-11T09:42:15.202 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-02-11T09:42:15.202 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2021-02-11T09:42:15.202 INFO:tasks.ceph:[client] rgw data log num shards = 4 2021-02-11T09:42:15.203 INFO:tasks.ceph:[client] rgw md log max shards = 4 2021-02-11T09:42:15.203 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2021-02-11T09:42:15.203 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2021-02-11T09:42:15.203 INFO:tasks.ceph:[client] setgroup = ceph 2021-02-11T09:42:15.203 INFO:tasks.ceph:[client] setuser = ceph 2021-02-11T09:42:15.203 INFO:tasks.ceph:[global] osd heartbeat grace = 40 2021-02-11T09:42:15.203 INFO:tasks.ceph:[mgr] debug mgr = 20 2021-02-11T09:42:15.203 INFO:tasks.ceph:[mgr] debug ms = 1 2021-02-11T09:42:15.203 INFO:tasks.ceph:[mon] debug mon = 20 2021-02-11T09:42:15.203 INFO:tasks.ceph:[mon] debug ms = 1 2021-02-11T09:42:15.203 INFO:tasks.ceph:[mon] debug paxos = 20 2021-02-11T09:42:15.203 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2021-02-11T09:42:15.204 INFO:tasks.ceph:[osd] debug filestore = 20 2021-02-11T09:42:15.204 INFO:tasks.ceph:[osd] debug journal = 20 2021-02-11T09:42:15.204 INFO:tasks.ceph:[osd] debug ms = 20 2021-02-11T09:42:15.204 INFO:tasks.ceph:[osd] debug osd = 25 2021-02-11T09:42:15.204 INFO:tasks.ceph:[osd] osd fast shutdown = False 2021-02-11T09:42:15.204 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2021-02-11T09:42:15.204 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2021-02-11T09:42:15.204 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2021-02-11T09:42:15.204 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2021-02-11T09:42:15.204 INFO:tasks.ceph:Setting up c1.mon.a... 2021-02-11T09:42:15.205 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c1.keyring 2021-02-11T09:42:15.275 INFO:teuthology.orchestra.run.smithi094.stdout:creating /etc/ceph/c1.keyring 2021-02-11T09:42:15.277 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c1.keyring 2021-02-11T09:42:15.353 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 0644 /etc/ceph/c1.keyring 2021-02-11T09:42:15.408 DEBUG:tasks.ceph:Ceph mon addresses: [('c1.mon.a', '172.21.15.94')] 2021-02-11T09:42:15.408 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.94', 'osd heartbeat grace': 40}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'mon osd crush smoke test': False}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2021-02-11T09:42:15.409 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2021-02-11T09:42:15.409 DEBUG:teuthology.orchestra.run.smithi094:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2021-02-11T09:42:15.457 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.94 --print /home/ubuntu/cephtest/c1.monmap 2021-02-11T09:42:15.531 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:42:15.532 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:42:15.532 INFO:teuthology.orchestra.run.smithi094.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c1.monmap 2021-02-11T09:42:15.532 INFO:teuthology.orchestra.run.smithi094.stdout:monmaptool: generated fsid 67a12f0c-a3f1-4f95-b0bb-3bdc01f1c8cf 2021-02-11T09:42:15.532 INFO:teuthology.orchestra.run.smithi094.stdout:epoch 0 2021-02-11T09:42:15.532 INFO:teuthology.orchestra.run.smithi094.stdout:fsid 67a12f0c-a3f1-4f95-b0bb-3bdc01f1c8cf 2021-02-11T09:42:15.532 INFO:teuthology.orchestra.run.smithi094.stdout:last_changed 2021-02-11T09:42:15.534716+0000 2021-02-11T09:42:15.532 INFO:teuthology.orchestra.run.smithi094.stdout:created 2021-02-11T09:42:15.534716+0000 2021-02-11T09:42:15.532 INFO:teuthology.orchestra.run.smithi094.stdout:min_mon_release 0 (unknown) 2021-02-11T09:42:15.533 INFO:teuthology.orchestra.run.smithi094.stdout:election_strategy: 1 2021-02-11T09:42:15.533 INFO:teuthology.orchestra.run.smithi094.stdout:0: [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] mon.a 2021-02-11T09:42:15.533 INFO:teuthology.orchestra.run.smithi094.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c1.monmap (1 monitors) 2021-02-11T09:42:15.534 DEBUG:teuthology.orchestra.run.smithi094:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2021-02-11T09:42:15.580 INFO:tasks.ceph:Writing /etc/ceph/c1.conf for FSID 67a12f0c-a3f1-4f95-b0bb-3bdc01f1c8cf... 2021-02-11T09:42:15.581 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2021-02-11T09:42:15.625 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2021-02-11T09:42:15.654 INFO:teuthology.orchestra.run.smithi094.stdout:[global] 2021-02-11T09:42:15.654 INFO:teuthology.orchestra.run.smithi094.stdout: chdir = "" 2021-02-11T09:42:15.654 INFO:teuthology.orchestra.run.smithi094.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-02-11T09:42:15.654 INFO:teuthology.orchestra.run.smithi094.stdout: auth supported = cephx 2021-02-11T09:42:15.654 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-02-11T09:42:15.654 INFO:teuthology.orchestra.run.smithi094.stdout: filestore xattr use omap = true 2021-02-11T09:42:15.654 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-02-11T09:42:15.654 INFO:teuthology.orchestra.run.smithi094.stdout: mon clock drift allowed = 1.000 2021-02-11T09:42:15.654 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-02-11T09:42:15.654 INFO:teuthology.orchestra.run.smithi094.stdout: osd crush chooseleaf type = 0 2021-02-11T09:42:15.655 INFO:teuthology.orchestra.run.smithi094.stdout: auth debug = true 2021-02-11T09:42:15.655 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-02-11T09:42:15.655 INFO:teuthology.orchestra.run.smithi094.stdout: ms die on old message = true 2021-02-11T09:42:15.655 INFO:teuthology.orchestra.run.smithi094.stdout: ms die on bug = true 2021-02-11T09:42:15.655 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-02-11T09:42:15.655 INFO:teuthology.orchestra.run.smithi094.stdout: mon max pg per osd = 10000 # >= luminous 2021-02-11T09:42:15.655 INFO:teuthology.orchestra.run.smithi094.stdout: mon pg warn max object skew = 0 2021-02-11T09:42:15.655 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-02-11T09:42:15.655 INFO:teuthology.orchestra.run.smithi094.stdout: # disable pg_autoscaler by default for new pools 2021-02-11T09:42:15.655 INFO:teuthology.orchestra.run.smithi094.stdout: osd_pool_default_pg_autoscale_mode = off 2021-02-11T09:42:15.655 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-02-11T09:42:15.655 INFO:teuthology.orchestra.run.smithi094.stdout: osd pool default size = 2 2021-02-11T09:42:15.655 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-02-11T09:42:15.656 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd allow primary affinity = true 2021-02-11T09:42:15.656 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd allow pg remap = true 2021-02-11T09:42:15.656 INFO:teuthology.orchestra.run.smithi094.stdout: mon warn on legacy crush tunables = false 2021-02-11T09:42:15.656 INFO:teuthology.orchestra.run.smithi094.stdout: mon warn on crush straw calc version zero = false 2021-02-11T09:42:15.656 INFO:teuthology.orchestra.run.smithi094.stdout: mon warn on no sortbitwise = false 2021-02-11T09:42:15.656 INFO:teuthology.orchestra.run.smithi094.stdout: mon warn on osd down out interval zero = false 2021-02-11T09:42:15.656 INFO:teuthology.orchestra.run.smithi094.stdout: mon warn on too few osds = false 2021-02-11T09:42:15.656 INFO:teuthology.orchestra.run.smithi094.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-02-11T09:42:15.656 INFO:teuthology.orchestra.run.smithi094.stdout: mon_warn_on_pool_no_redundancy = false 2021-02-11T09:42:15.656 INFO:teuthology.orchestra.run.smithi094.stdout: mon_allow_pool_size_one = true 2021-02-11T09:42:15.656 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-02-11T09:42:15.656 INFO:teuthology.orchestra.run.smithi094.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-02-11T09:42:15.656 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-02-11T09:42:15.656 INFO:teuthology.orchestra.run.smithi094.stdout: osd default data pool replay window = 5 2021-02-11T09:42:15.656 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-02-11T09:42:15.657 INFO:teuthology.orchestra.run.smithi094.stdout: mon allow pool delete = true 2021-02-11T09:42:15.657 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-02-11T09:42:15.657 INFO:teuthology.orchestra.run.smithi094.stdout: mon cluster log file level = debug 2021-02-11T09:42:15.657 INFO:teuthology.orchestra.run.smithi094.stdout: debug asserts on shutdown = true 2021-02-11T09:42:15.657 INFO:teuthology.orchestra.run.smithi094.stdout: mon health detail to clog = false 2021-02-11T09:42:15.657 INFO:teuthology.orchestra.run.smithi094.stdout: mon host = 172.21.15.94 2021-02-11T09:42:15.657 INFO:teuthology.orchestra.run.smithi094.stdout: osd heartbeat grace = 40 2021-02-11T09:42:15.657 INFO:teuthology.orchestra.run.smithi094.stdout: fsid = 67a12f0c-a3f1-4f95-b0bb-3bdc01f1c8cf 2021-02-11T09:42:15.657 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-02-11T09:42:15.657 INFO:teuthology.orchestra.run.smithi094.stdout:[osd] 2021-02-11T09:42:15.657 INFO:teuthology.orchestra.run.smithi094.stdout: osd journal size = 100 2021-02-11T09:42:15.657 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-02-11T09:42:15.657 INFO:teuthology.orchestra.run.smithi094.stdout: osd scrub load threshold = 5.0 2021-02-11T09:42:15.658 INFO:teuthology.orchestra.run.smithi094.stdout: osd scrub max interval = 600 2021-02-11T09:42:15.658 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-02-11T09:42:15.658 INFO:teuthology.orchestra.run.smithi094.stdout: osd recover clone overlap = true 2021-02-11T09:42:15.658 INFO:teuthology.orchestra.run.smithi094.stdout: osd recovery max chunk = 1048576 2021-02-11T09:42:15.658 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-02-11T09:42:15.658 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug shutdown = true 2021-02-11T09:42:15.658 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug op order = true 2021-02-11T09:42:15.658 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug verify stray on activate = true 2021-02-11T09:42:15.658 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-02-11T09:42:15.658 INFO:teuthology.orchestra.run.smithi094.stdout: osd open classes on start = true 2021-02-11T09:42:15.658 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug pg log writeout = true 2021-02-11T09:42:15.658 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-02-11T09:42:15.658 INFO:teuthology.orchestra.run.smithi094.stdout: osd deep scrub update digest min age = 30 2021-02-11T09:42:15.659 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-02-11T09:42:15.659 INFO:teuthology.orchestra.run.smithi094.stdout: osd map max advance = 10 2021-02-11T09:42:15.659 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-02-11T09:42:15.659 INFO:teuthology.orchestra.run.smithi094.stdout: journal zero on create = true 2021-02-11T09:42:15.659 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-02-11T09:42:15.659 INFO:teuthology.orchestra.run.smithi094.stdout: filestore ondisk finisher threads = 3 2021-02-11T09:42:15.659 INFO:teuthology.orchestra.run.smithi094.stdout: filestore apply finisher threads = 3 2021-02-11T09:42:15.659 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-02-11T09:42:15.659 INFO:teuthology.orchestra.run.smithi094.stdout: bdev debug aio = true 2021-02-11T09:42:15.659 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug misdirected ops = true 2021-02-11T09:42:15.659 INFO:teuthology.orchestra.run.smithi094.stdout: debug filestore = 20 2021-02-11T09:42:15.659 INFO:teuthology.orchestra.run.smithi094.stdout: debug journal = 20 2021-02-11T09:42:15.659 INFO:teuthology.orchestra.run.smithi094.stdout: debug ms = 20 2021-02-11T09:42:15.660 INFO:teuthology.orchestra.run.smithi094.stdout: debug osd = 25 2021-02-11T09:42:15.660 INFO:teuthology.orchestra.run.smithi094.stdout: osd fast shutdown = False 2021-02-11T09:42:15.660 INFO:teuthology.orchestra.run.smithi094.stdout: rocksdb delete range threshold = 0 2021-02-11T09:42:15.660 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-02-11T09:42:15.660 INFO:teuthology.orchestra.run.smithi094.stdout:[mgr] 2021-02-11T09:42:15.660 INFO:teuthology.orchestra.run.smithi094.stdout: debug ms = 1 2021-02-11T09:42:15.660 INFO:teuthology.orchestra.run.smithi094.stdout: debug mgr = 20 2021-02-11T09:42:15.660 INFO:teuthology.orchestra.run.smithi094.stdout: debug mon = 20 2021-02-11T09:42:15.660 INFO:teuthology.orchestra.run.smithi094.stdout: debug auth = 20 2021-02-11T09:42:15.660 INFO:teuthology.orchestra.run.smithi094.stdout: mon reweight min pgs per osd = 4 2021-02-11T09:42:15.660 INFO:teuthology.orchestra.run.smithi094.stdout: mon reweight min bytes per osd = 10 2021-02-11T09:42:15.660 INFO:teuthology.orchestra.run.smithi094.stdout: mgr/telemetry/nag = false 2021-02-11T09:42:15.660 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-02-11T09:42:15.660 INFO:teuthology.orchestra.run.smithi094.stdout:[mon] 2021-02-11T09:42:15.661 INFO:teuthology.orchestra.run.smithi094.stdout: debug ms = 1 2021-02-11T09:42:15.661 INFO:teuthology.orchestra.run.smithi094.stdout: debug mon = 20 2021-02-11T09:42:15.661 INFO:teuthology.orchestra.run.smithi094.stdout: debug paxos = 20 2021-02-11T09:42:15.661 INFO:teuthology.orchestra.run.smithi094.stdout: debug auth = 20 2021-02-11T09:42:15.661 INFO:teuthology.orchestra.run.smithi094.stdout: mon data avail warn = 5 2021-02-11T09:42:15.661 INFO:teuthology.orchestra.run.smithi094.stdout: mon mgr mkfs grace = 240 2021-02-11T09:42:15.661 INFO:teuthology.orchestra.run.smithi094.stdout: mon reweight min pgs per osd = 4 2021-02-11T09:42:15.661 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd reporter subtree level = osd 2021-02-11T09:42:15.661 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd prime pg temp = true 2021-02-11T09:42:15.661 INFO:teuthology.orchestra.run.smithi094.stdout: mon reweight min bytes per osd = 10 2021-02-11T09:42:15.661 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd crush smoke test = False 2021-02-11T09:42:15.661 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-02-11T09:42:15.661 INFO:teuthology.orchestra.run.smithi094.stdout:[client] 2021-02-11T09:42:15.661 INFO:teuthology.orchestra.run.smithi094.stdout: rgw cache enabled = true 2021-02-11T09:42:15.661 INFO:teuthology.orchestra.run.smithi094.stdout: rgw enable ops log = true 2021-02-11T09:42:15.661 INFO:teuthology.orchestra.run.smithi094.stdout: rgw enable usage log = true 2021-02-11T09:42:15.662 INFO:teuthology.orchestra.run.smithi094.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-02-11T09:42:15.662 INFO:teuthology.orchestra.run.smithi094.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-02-11T09:42:15.662 INFO:teuthology.orchestra.run.smithi094.stdout: debug rgw = 20 2021-02-11T09:42:15.662 INFO:teuthology.orchestra.run.smithi094.stdout: rgw crypt require ssl = False 2021-02-11T09:42:15.662 INFO:teuthology.orchestra.run.smithi094.stdout: rgw crypt s3 kms backend = testing 2021-02-11T09:42:15.662 INFO:teuthology.orchestra.run.smithi094.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-02-11T09:42:15.662 INFO:teuthology.orchestra.run.smithi094.stdout: rgw curl low speed time = 300 2021-02-11T09:42:15.662 INFO:teuthology.orchestra.run.smithi094.stdout: rgw data log num shards = 4 2021-02-11T09:42:15.662 INFO:teuthology.orchestra.run.smithi094.stdout: rgw md log max shards = 4 2021-02-11T09:42:15.662 INFO:teuthology.orchestra.run.smithi094.stdout: rgw sync log trim interval = 0 2021-02-11T09:42:15.662 INFO:teuthology.orchestra.run.smithi094.stdout: rgw sync obj etag verify = True 2021-02-11T09:42:15.662 INFO:teuthology.orchestra.run.smithi094.stdout: setgroup = ceph 2021-02-11T09:42:15.662 INFO:teuthology.orchestra.run.smithi094.stdout: setuser = ceph 2021-02-11T09:42:15.663 INFO:teuthology.orchestra.run.smithi094.stdout:[mon.a] 2021-02-11T09:42:15.663 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.0] 2021-02-11T09:42:15.663 INFO:teuthology.orchestra.run.smithi094.stdout: osd_max_omap_entries_per_request = 10 2021-02-11T09:42:15.663 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.1] 2021-02-11T09:42:15.663 INFO:teuthology.orchestra.run.smithi094.stdout: osd_max_omap_entries_per_request = 1000 2021-02-11T09:42:15.663 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.2] 2021-02-11T09:42:15.663 INFO:teuthology.orchestra.run.smithi094.stdout: osd_max_omap_entries_per_request = 10000 2021-02-11T09:42:15.664 INFO:teuthology.orchestra.run.smithi098.stdout:[global] 2021-02-11T09:42:15.664 INFO:teuthology.orchestra.run.smithi098.stdout: chdir = "" 2021-02-11T09:42:15.664 INFO:teuthology.orchestra.run.smithi098.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-02-11T09:42:15.664 INFO:teuthology.orchestra.run.smithi098.stdout: auth supported = cephx 2021-02-11T09:42:15.664 INFO:teuthology.orchestra.run.smithi098.stdout: 2021-02-11T09:42:15.664 INFO:teuthology.orchestra.run.smithi098.stdout: filestore xattr use omap = true 2021-02-11T09:42:15.664 INFO:teuthology.orchestra.run.smithi098.stdout: 2021-02-11T09:42:15.664 INFO:teuthology.orchestra.run.smithi098.stdout: mon clock drift allowed = 1.000 2021-02-11T09:42:15.664 INFO:teuthology.orchestra.run.smithi098.stdout: 2021-02-11T09:42:15.665 INFO:teuthology.orchestra.run.smithi098.stdout: osd crush chooseleaf type = 0 2021-02-11T09:42:15.665 INFO:teuthology.orchestra.run.smithi098.stdout: auth debug = true 2021-02-11T09:42:15.665 INFO:teuthology.orchestra.run.smithi098.stdout: 2021-02-11T09:42:15.665 INFO:teuthology.orchestra.run.smithi098.stdout: ms die on old message = true 2021-02-11T09:42:15.665 INFO:teuthology.orchestra.run.smithi098.stdout: ms die on bug = true 2021-02-11T09:42:15.665 INFO:teuthology.orchestra.run.smithi098.stdout: 2021-02-11T09:42:15.665 INFO:teuthology.orchestra.run.smithi098.stdout: mon max pg per osd = 10000 # >= luminous 2021-02-11T09:42:15.665 INFO:teuthology.orchestra.run.smithi098.stdout: mon pg warn max object skew = 0 2021-02-11T09:42:15.665 INFO:teuthology.orchestra.run.smithi098.stdout: 2021-02-11T09:42:15.665 INFO:teuthology.orchestra.run.smithi098.stdout: # disable pg_autoscaler by default for new pools 2021-02-11T09:42:15.665 INFO:teuthology.orchestra.run.smithi098.stdout: osd_pool_default_pg_autoscale_mode = off 2021-02-11T09:42:15.665 INFO:teuthology.orchestra.run.smithi098.stdout: 2021-02-11T09:42:15.665 INFO:teuthology.orchestra.run.smithi098.stdout: osd pool default size = 2 2021-02-11T09:42:15.665 INFO:teuthology.orchestra.run.smithi098.stdout: 2021-02-11T09:42:15.665 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd allow primary affinity = true 2021-02-11T09:42:15.666 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd allow pg remap = true 2021-02-11T09:42:15.666 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on legacy crush tunables = false 2021-02-11T09:42:15.666 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on crush straw calc version zero = false 2021-02-11T09:42:15.666 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on no sortbitwise = false 2021-02-11T09:42:15.666 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on osd down out interval zero = false 2021-02-11T09:42:15.666 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on too few osds = false 2021-02-11T09:42:15.666 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-02-11T09:42:15.666 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_pool_no_redundancy = false 2021-02-11T09:42:15.666 INFO:teuthology.orchestra.run.smithi098.stdout: mon_allow_pool_size_one = true 2021-02-11T09:42:15.666 INFO:teuthology.orchestra.run.smithi098.stdout: 2021-02-11T09:42:15.666 INFO:teuthology.orchestra.run.smithi098.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-02-11T09:42:15.666 INFO:teuthology.orchestra.run.smithi098.stdout: 2021-02-11T09:42:15.667 INFO:teuthology.orchestra.run.smithi098.stdout: osd default data pool replay window = 5 2021-02-11T09:42:15.667 INFO:teuthology.orchestra.run.smithi098.stdout: 2021-02-11T09:42:15.667 INFO:teuthology.orchestra.run.smithi098.stdout: mon allow pool delete = true 2021-02-11T09:42:15.667 INFO:teuthology.orchestra.run.smithi098.stdout: 2021-02-11T09:42:15.667 INFO:teuthology.orchestra.run.smithi098.stdout: mon cluster log file level = debug 2021-02-11T09:42:15.667 INFO:teuthology.orchestra.run.smithi098.stdout: debug asserts on shutdown = true 2021-02-11T09:42:15.667 INFO:teuthology.orchestra.run.smithi098.stdout: mon health detail to clog = false 2021-02-11T09:42:15.667 INFO:teuthology.orchestra.run.smithi098.stdout: mon host = 172.21.15.94 2021-02-11T09:42:15.667 INFO:teuthology.orchestra.run.smithi098.stdout: osd heartbeat grace = 40 2021-02-11T09:42:15.667 INFO:teuthology.orchestra.run.smithi098.stdout: fsid = 67a12f0c-a3f1-4f95-b0bb-3bdc01f1c8cf 2021-02-11T09:42:15.667 INFO:teuthology.orchestra.run.smithi098.stdout: 2021-02-11T09:42:15.667 INFO:teuthology.orchestra.run.smithi098.stdout:[osd] 2021-02-11T09:42:15.667 INFO:teuthology.orchestra.run.smithi098.stdout: osd journal size = 100 2021-02-11T09:42:15.667 INFO:teuthology.orchestra.run.smithi098.stdout: 2021-02-11T09:42:15.668 INFO:teuthology.orchestra.run.smithi098.stdout: osd scrub load threshold = 5.0 2021-02-11T09:42:15.668 INFO:teuthology.orchestra.run.smithi098.stdout: osd scrub max interval = 600 2021-02-11T09:42:15.668 INFO:teuthology.orchestra.run.smithi098.stdout: 2021-02-11T09:42:15.668 INFO:teuthology.orchestra.run.smithi098.stdout: osd recover clone overlap = true 2021-02-11T09:42:15.668 INFO:teuthology.orchestra.run.smithi098.stdout: osd recovery max chunk = 1048576 2021-02-11T09:42:15.668 INFO:teuthology.orchestra.run.smithi098.stdout: 2021-02-11T09:42:15.668 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug shutdown = true 2021-02-11T09:42:15.668 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug op order = true 2021-02-11T09:42:15.668 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug verify stray on activate = true 2021-02-11T09:42:15.668 INFO:teuthology.orchestra.run.smithi098.stdout: 2021-02-11T09:42:15.668 INFO:teuthology.orchestra.run.smithi098.stdout: osd open classes on start = true 2021-02-11T09:42:15.668 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug pg log writeout = true 2021-02-11T09:42:15.668 INFO:teuthology.orchestra.run.smithi098.stdout: 2021-02-11T09:42:15.668 INFO:teuthology.orchestra.run.smithi098.stdout: osd deep scrub update digest min age = 30 2021-02-11T09:42:15.669 INFO:teuthology.orchestra.run.smithi098.stdout: 2021-02-11T09:42:15.669 INFO:teuthology.orchestra.run.smithi098.stdout: osd map max advance = 10 2021-02-11T09:42:15.669 INFO:teuthology.orchestra.run.smithi098.stdout: 2021-02-11T09:42:15.669 INFO:teuthology.orchestra.run.smithi098.stdout: journal zero on create = true 2021-02-11T09:42:15.669 INFO:teuthology.orchestra.run.smithi098.stdout: 2021-02-11T09:42:15.669 INFO:teuthology.orchestra.run.smithi098.stdout: filestore ondisk finisher threads = 3 2021-02-11T09:42:15.669 INFO:teuthology.orchestra.run.smithi098.stdout: filestore apply finisher threads = 3 2021-02-11T09:42:15.669 INFO:teuthology.orchestra.run.smithi098.stdout: 2021-02-11T09:42:15.670 INFO:teuthology.orchestra.run.smithi098.stdout: bdev debug aio = true 2021-02-11T09:42:15.670 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug misdirected ops = true 2021-02-11T09:42:15.670 INFO:teuthology.orchestra.run.smithi098.stdout: debug filestore = 20 2021-02-11T09:42:15.670 INFO:teuthology.orchestra.run.smithi098.stdout: debug journal = 20 2021-02-11T09:42:15.670 INFO:teuthology.orchestra.run.smithi098.stdout: debug ms = 20 2021-02-11T09:42:15.670 INFO:teuthology.orchestra.run.smithi098.stdout: debug osd = 25 2021-02-11T09:42:15.670 INFO:teuthology.orchestra.run.smithi098.stdout: osd fast shutdown = False 2021-02-11T09:42:15.670 INFO:teuthology.orchestra.run.smithi098.stdout: rocksdb delete range threshold = 0 2021-02-11T09:42:15.670 INFO:teuthology.orchestra.run.smithi098.stdout: 2021-02-11T09:42:15.670 INFO:teuthology.orchestra.run.smithi098.stdout:[mgr] 2021-02-11T09:42:15.670 INFO:teuthology.orchestra.run.smithi098.stdout: debug ms = 1 2021-02-11T09:42:15.670 INFO:teuthology.orchestra.run.smithi098.stdout: debug mgr = 20 2021-02-11T09:42:15.671 INFO:teuthology.orchestra.run.smithi098.stdout: debug mon = 20 2021-02-11T09:42:15.671 INFO:teuthology.orchestra.run.smithi098.stdout: debug auth = 20 2021-02-11T09:42:15.671 INFO:teuthology.orchestra.run.smithi098.stdout: mon reweight min pgs per osd = 4 2021-02-11T09:42:15.671 INFO:teuthology.orchestra.run.smithi098.stdout: mon reweight min bytes per osd = 10 2021-02-11T09:42:15.671 INFO:teuthology.orchestra.run.smithi098.stdout: mgr/telemetry/nag = false 2021-02-11T09:42:15.671 INFO:teuthology.orchestra.run.smithi098.stdout: 2021-02-11T09:42:15.671 INFO:teuthology.orchestra.run.smithi098.stdout:[mon] 2021-02-11T09:42:15.671 INFO:teuthology.orchestra.run.smithi098.stdout: debug ms = 1 2021-02-11T09:42:15.671 INFO:teuthology.orchestra.run.smithi098.stdout: debug mon = 20 2021-02-11T09:42:15.671 INFO:teuthology.orchestra.run.smithi098.stdout: debug paxos = 20 2021-02-11T09:42:15.671 INFO:teuthology.orchestra.run.smithi098.stdout: debug auth = 20 2021-02-11T09:42:15.671 INFO:teuthology.orchestra.run.smithi098.stdout: mon data avail warn = 5 2021-02-11T09:42:15.671 INFO:teuthology.orchestra.run.smithi098.stdout: mon mgr mkfs grace = 240 2021-02-11T09:42:15.671 INFO:teuthology.orchestra.run.smithi098.stdout: mon reweight min pgs per osd = 4 2021-02-11T09:42:15.671 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd reporter subtree level = osd 2021-02-11T09:42:15.672 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd prime pg temp = true 2021-02-11T09:42:15.672 INFO:teuthology.orchestra.run.smithi098.stdout: mon reweight min bytes per osd = 10 2021-02-11T09:42:15.672 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd crush smoke test = False 2021-02-11T09:42:15.672 INFO:teuthology.orchestra.run.smithi098.stdout: 2021-02-11T09:42:15.672 INFO:teuthology.orchestra.run.smithi098.stdout:[client] 2021-02-11T09:42:15.672 INFO:teuthology.orchestra.run.smithi098.stdout: rgw cache enabled = true 2021-02-11T09:42:15.672 INFO:teuthology.orchestra.run.smithi098.stdout: rgw enable ops log = true 2021-02-11T09:42:15.672 INFO:teuthology.orchestra.run.smithi098.stdout: rgw enable usage log = true 2021-02-11T09:42:15.672 INFO:teuthology.orchestra.run.smithi098.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-02-11T09:42:15.672 INFO:teuthology.orchestra.run.smithi098.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-02-11T09:42:15.672 INFO:teuthology.orchestra.run.smithi098.stdout: debug rgw = 20 2021-02-11T09:42:15.672 INFO:teuthology.orchestra.run.smithi098.stdout: rgw crypt require ssl = False 2021-02-11T09:42:15.672 INFO:teuthology.orchestra.run.smithi098.stdout: rgw crypt s3 kms backend = testing 2021-02-11T09:42:15.672 INFO:teuthology.orchestra.run.smithi098.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-02-11T09:42:15.673 INFO:teuthology.orchestra.run.smithi098.stdout: rgw curl low speed time = 300 2021-02-11T09:42:15.673 INFO:teuthology.orchestra.run.smithi098.stdout: rgw data log num shards = 4 2021-02-11T09:42:15.673 INFO:teuthology.orchestra.run.smithi098.stdout: rgw md log max shards = 4 2021-02-11T09:42:15.673 INFO:teuthology.orchestra.run.smithi098.stdout: rgw sync log trim interval = 0 2021-02-11T09:42:15.673 INFO:teuthology.orchestra.run.smithi098.stdout: rgw sync obj etag verify = True 2021-02-11T09:42:15.673 INFO:teuthology.orchestra.run.smithi098.stdout: setgroup = ceph 2021-02-11T09:42:15.673 INFO:teuthology.orchestra.run.smithi098.stdout: setuser = ceph 2021-02-11T09:42:15.673 INFO:teuthology.orchestra.run.smithi098.stdout:[mon.a] 2021-02-11T09:42:15.673 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.0] 2021-02-11T09:42:15.673 INFO:teuthology.orchestra.run.smithi098.stdout: osd_max_omap_entries_per_request = 10 2021-02-11T09:42:15.673 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.1] 2021-02-11T09:42:15.673 INFO:teuthology.orchestra.run.smithi098.stdout: osd_max_omap_entries_per_request = 1000 2021-02-11T09:42:15.673 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.2] 2021-02-11T09:42:15.673 INFO:teuthology.orchestra.run.smithi098.stdout: osd_max_omap_entries_per_request = 10000 2021-02-11T09:42:15.675 INFO:tasks.ceph:Creating admin key on c1.mon.a... 2021-02-11T09:42:15.675 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c1.keyring 2021-02-11T09:42:15.752 INFO:tasks.ceph:Copying monmap to all nodes... 2021-02-11T09:42:15.752 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2021-02-11T09:42:15.752 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/etc/ceph/c1.keyring of=/dev/stdout 2021-02-11T09:42:15.801 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2021-02-11T09:42:15.801 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/home/ubuntu/cephtest/c1.monmap of=/dev/stdout 2021-02-11T09:42:15.848 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi094.front.sepia.ceph.com 2021-02-11T09:42:15.848 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2021-02-11T09:42:15.849 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/etc/ceph/c1.keyring 2021-02-11T09:42:15.849 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 0644 /etc/ceph/c1.keyring 2021-02-11T09:42:15.915 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2021-02-11T09:42:15.915 DEBUG:teuthology.orchestra.run.smithi094:> dd of=/home/ubuntu/cephtest/c1.monmap 2021-02-11T09:42:15.961 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi098.front.sepia.ceph.com 2021-02-11T09:42:15.961 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2021-02-11T09:42:15.961 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/ceph/c1.keyring 2021-02-11T09:42:15.961 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0644 /etc/ceph/c1.keyring 2021-02-11T09:42:15.983 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2021-02-11T09:42:15.984 DEBUG:teuthology.orchestra.run.smithi098:> dd of=/home/ubuntu/cephtest/c1.monmap 2021-02-11T09:42:16.033 INFO:tasks.ceph:Setting up mon nodes... 2021-02-11T09:42:16.033 INFO:tasks.ceph:Setting up mgr nodes... 2021-02-11T09:42:16.033 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /var/lib/ceph/mgr/c1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c1-x/keyring 2021-02-11T09:42:16.076 INFO:teuthology.orchestra.run.smithi094.stdout:creating /var/lib/ceph/mgr/c1-x/keyring 2021-02-11T09:42:16.078 INFO:tasks.ceph:Setting up mds nodes... 2021-02-11T09:42:16.078 INFO:tasks.ceph_client:Setting up client nodes... 2021-02-11T09:42:16.078 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c1.client.0.keyring && sudo chmod 0644 /etc/ceph/c1.client.0.keyring 2021-02-11T09:42:16.151 INFO:teuthology.orchestra.run.smithi094.stdout:creating /etc/ceph/c1.client.0.keyring 2021-02-11T09:42:16.163 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c1.client.1.keyring && sudo chmod 0644 /etc/ceph/c1.client.1.keyring 2021-02-11T09:42:16.239 INFO:teuthology.orchestra.run.smithi094.stdout:creating /etc/ceph/c1.client.1.keyring 2021-02-11T09:42:16.252 INFO:tasks.ceph:Running mkfs on osd nodes... 2021-02-11T09:42:16.252 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi094.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2021-02-11T09:42:16.252 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /var/lib/ceph/osd/c1-0 2021-02-11T09:42:16.309 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2021-02-11T09:42:16.310 INFO:tasks.ceph:role: c1.osd.0 2021-02-11T09:42:16.310 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi094.front.sepia.ceph.com 2021-02-11T09:42:16.310 DEBUG:teuthology.orchestra.run.smithi094:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2021-02-11T09:42:16.390 INFO:teuthology.orchestra.run.smithi094.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2021-02-11T09:42:16.390 INFO:teuthology.orchestra.run.smithi094.stdout: = sectsz=512 attr=2, projid32bit=1 2021-02-11T09:42:16.390 INFO:teuthology.orchestra.run.smithi094.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2021-02-11T09:42:16.390 INFO:teuthology.orchestra.run.smithi094.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-02-11T09:42:16.390 INFO:teuthology.orchestra.run.smithi094.stdout: = sunit=0 swidth=0 blks 2021-02-11T09:42:16.390 INFO:teuthology.orchestra.run.smithi094.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-02-11T09:42:16.390 INFO:teuthology.orchestra.run.smithi094.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-02-11T09:42:16.390 INFO:teuthology.orchestra.run.smithi094.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-02-11T09:42:16.390 INFO:teuthology.orchestra.run.smithi094.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-02-11T09:42:16.391 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi094.front.sepia.ceph.com -o noatime 2021-02-11T09:42:16.391 DEBUG:teuthology.orchestra.run.smithi094:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c1-0 2021-02-11T09:42:16.455 DEBUG:teuthology.orchestra.run.smithi094:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-0 2021-02-11T09:42:16.506 INFO:teuthology.orchestra.run.smithi094.stderr:sudo: /sbin/restorecon: command not found 2021-02-11T09:42:16.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-11T09:42:16.507 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /var/lib/ceph/osd/c1-1 2021-02-11T09:42:16.562 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2021-02-11T09:42:16.562 INFO:tasks.ceph:role: c1.osd.1 2021-02-11T09:42:16.562 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi094.front.sepia.ceph.com 2021-02-11T09:42:16.562 DEBUG:teuthology.orchestra.run.smithi094:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2021-02-11T09:42:16.643 INFO:teuthology.orchestra.run.smithi094.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2021-02-11T09:42:16.643 INFO:teuthology.orchestra.run.smithi094.stdout: = sectsz=512 attr=2, projid32bit=1 2021-02-11T09:42:16.643 INFO:teuthology.orchestra.run.smithi094.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2021-02-11T09:42:16.643 INFO:teuthology.orchestra.run.smithi094.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-02-11T09:42:16.643 INFO:teuthology.orchestra.run.smithi094.stdout: = sunit=0 swidth=0 blks 2021-02-11T09:42:16.643 INFO:teuthology.orchestra.run.smithi094.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-02-11T09:42:16.643 INFO:teuthology.orchestra.run.smithi094.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-02-11T09:42:16.643 INFO:teuthology.orchestra.run.smithi094.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-02-11T09:42:16.643 INFO:teuthology.orchestra.run.smithi094.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-02-11T09:42:16.644 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi094.front.sepia.ceph.com -o noatime 2021-02-11T09:42:16.644 DEBUG:teuthology.orchestra.run.smithi094:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c1-1 2021-02-11T09:42:16.708 DEBUG:teuthology.orchestra.run.smithi094:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-1 2021-02-11T09:42:16.762 INFO:teuthology.orchestra.run.smithi094.stderr:sudo: /sbin/restorecon: command not found 2021-02-11T09:42:16.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-11T09:42:16.763 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /var/lib/ceph/osd/c1-2 2021-02-11T09:42:16.818 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2021-02-11T09:42:16.818 INFO:tasks.ceph:role: c1.osd.2 2021-02-11T09:42:16.818 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi094.front.sepia.ceph.com 2021-02-11T09:42:16.818 DEBUG:teuthology.orchestra.run.smithi094:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2021-02-11T09:42:16.899 INFO:teuthology.orchestra.run.smithi094.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2021-02-11T09:42:16.899 INFO:teuthology.orchestra.run.smithi094.stdout: = sectsz=512 attr=2, projid32bit=1 2021-02-11T09:42:16.899 INFO:teuthology.orchestra.run.smithi094.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2021-02-11T09:42:16.900 INFO:teuthology.orchestra.run.smithi094.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-02-11T09:42:16.900 INFO:teuthology.orchestra.run.smithi094.stdout: = sunit=0 swidth=0 blks 2021-02-11T09:42:16.900 INFO:teuthology.orchestra.run.smithi094.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-02-11T09:42:16.900 INFO:teuthology.orchestra.run.smithi094.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-02-11T09:42:16.900 INFO:teuthology.orchestra.run.smithi094.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-02-11T09:42:16.900 INFO:teuthology.orchestra.run.smithi094.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-02-11T09:42:16.901 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi094.front.sepia.ceph.com -o noatime 2021-02-11T09:42:16.901 DEBUG:teuthology.orchestra.run.smithi094:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c1-2 2021-02-11T09:42:16.963 DEBUG:teuthology.orchestra.run.smithi094:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-2 2021-02-11T09:42:17.014 INFO:teuthology.orchestra.run.smithi094.stderr:sudo: /sbin/restorecon: command not found 2021-02-11T09:42:17.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-11T09:42:17.015 DEBUG:teuthology.orchestra.run.smithi094:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c1.monmap 2021-02-11T09:42:17.102 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:42:17.099+0000 7fbc8c821080 -1 auth: error reading file: /var/lib/ceph/osd/c1-0/keyring: can't open /var/lib/ceph/osd/c1-0/keyring: (2) No such file or directory 2021-02-11T09:42:17.102 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:42:17.099+0000 7fbc8c821080 -1 created new key in keyring /var/lib/ceph/osd/c1-0/keyring 2021-02-11T09:42:17.103 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:42:17.099+0000 7fbc8c821080 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2021-02-11T09:42:17.103 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:42:17.099+0000 7fbc8c821080 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2021-02-11T09:42:17.103 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:42:17.099+0000 7fbc8c821080 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2021-02-11T09:42:17.103 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:42:17.099+0000 7fbc8c821080 -1 bluestore(/var/lib/ceph/osd/c1-0) _read_fsid unparsable uuid 2021-02-11T09:42:19.284 DEBUG:teuthology.orchestra.run.smithi094:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-0 2021-02-11T09:42:19.343 DEBUG:teuthology.orchestra.run.smithi094:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c1.monmap 2021-02-11T09:42:19.421 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:42:19.419+0000 7fc3cb3bf080 -1 auth: error reading file: /var/lib/ceph/osd/c1-1/keyring: can't open /var/lib/ceph/osd/c1-1/keyring: (2) No such file or directory 2021-02-11T09:42:19.421 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:42:19.419+0000 7fc3cb3bf080 -1 created new key in keyring /var/lib/ceph/osd/c1-1/keyring 2021-02-11T09:42:19.421 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:42:19.419+0000 7fc3cb3bf080 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2021-02-11T09:42:19.421 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:42:19.419+0000 7fc3cb3bf080 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2021-02-11T09:42:19.422 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:42:19.419+0000 7fc3cb3bf080 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2021-02-11T09:42:19.422 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:42:19.419+0000 7fc3cb3bf080 -1 bluestore(/var/lib/ceph/osd/c1-1) _read_fsid unparsable uuid 2021-02-11T09:42:21.585 DEBUG:teuthology.orchestra.run.smithi094:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-1 2021-02-11T09:42:21.643 DEBUG:teuthology.orchestra.run.smithi094:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c1.monmap 2021-02-11T09:42:21.721 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:42:21.719+0000 7ffbed32e080 -1 auth: error reading file: /var/lib/ceph/osd/c1-2/keyring: can't open /var/lib/ceph/osd/c1-2/keyring: (2) No such file or directory 2021-02-11T09:42:21.721 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:42:21.719+0000 7ffbed32e080 -1 created new key in keyring /var/lib/ceph/osd/c1-2/keyring 2021-02-11T09:42:21.721 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:42:21.719+0000 7ffbed32e080 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2021-02-11T09:42:21.722 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:42:21.719+0000 7ffbed32e080 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2021-02-11T09:42:21.722 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:42:21.719+0000 7ffbed32e080 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2021-02-11T09:42:21.722 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:42:21.719+0000 7ffbed32e080 -1 bluestore(/var/lib/ceph/osd/c1-2) _read_fsid unparsable uuid 2021-02-11T09:42:23.886 DEBUG:teuthology.orchestra.run.smithi094:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-2 2021-02-11T09:42:23.942 INFO:tasks.ceph:Reading keys from all nodes... 2021-02-11T09:42:23.942 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2021-02-11T09:42:23.943 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/mgr/c1-x/keyring of=/dev/stdout 2021-02-11T09:42:23.997 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2021-02-11T09:42:23.997 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/osd/c1-0/keyring of=/dev/stdout 2021-02-11T09:42:24.053 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2021-02-11T09:42:24.053 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/osd/c1-1/keyring of=/dev/stdout 2021-02-11T09:42:24.109 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2021-02-11T09:42:24.109 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/osd/c1-2/keyring of=/dev/stdout 2021-02-11T09:42:24.165 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2021-02-11T09:42:24.165 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/etc/ceph/c1.client.0.keyring of=/dev/stdout 2021-02-11T09:42:24.213 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2021-02-11T09:42:24.213 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/etc/ceph/c1.client.1.keyring of=/dev/stdout 2021-02-11T09:42:24.260 INFO:tasks.ceph:Adding keys to all mons... 2021-02-11T09:42:24.260 DEBUG:teuthology.orchestra.run.smithi094:> sudo tee -a /etc/ceph/c1.keyring 2021-02-11T09:42:24.316 INFO:teuthology.orchestra.run.smithi094.stdout:[mgr.x] 2021-02-11T09:42:24.316 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQD4+yRgi7OvBBAAzO4gllWUoeo7VRjqlQsp1w== 2021-02-11T09:42:24.316 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.0] 2021-02-11T09:42:24.316 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQD5+yRgRtU7BhAAzHEX6ECN+bpbVEJimbtcWw== 2021-02-11T09:42:24.316 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.1] 2021-02-11T09:42:24.316 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQD7+yRgMftAGRAAWM4g05CSapsF82I4yMascA== 2021-02-11T09:42:24.316 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.2] 2021-02-11T09:42:24.317 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQD9+yRgyRUkKxAA/vj0CZoPwYJpxbkpHoxaaA== 2021-02-11T09:42:24.317 INFO:teuthology.orchestra.run.smithi094.stdout:[client.0] 2021-02-11T09:42:24.317 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQD4+yRgaKEzCRAAwqaJeUI9KxLtXVlM0JYFzw== 2021-02-11T09:42:24.317 INFO:teuthology.orchestra.run.smithi094.stdout:[client.1] 2021-02-11T09:42:24.317 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQD4+yRgHuJxDhAAohGH7eQvoW7LJuCMKjTeQA== 2021-02-11T09:42:24.318 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2021-02-11T09:42:24.400 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-11T09:42:24.483 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-11T09:42:24.568 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-11T09:42:24.652 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-02-11T09:42:24.740 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-02-11T09:42:24.827 INFO:tasks.ceph:Running mkfs on mon nodes... 2021-02-11T09:42:24.828 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /var/lib/ceph/mon/c1-a 2021-02-11T09:42:24.885 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c1 --mkfs -i a --monmap /home/ubuntu/cephtest/c1.monmap --keyring /etc/ceph/c1.keyring 2021-02-11T09:42:24.983 DEBUG:teuthology.orchestra.run.smithi094:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c1-a 2021-02-11T09:42:24.997 DEBUG:teuthology.orchestra.run.smithi094:> rm -- /home/ubuntu/cephtest/c1.monmap 2021-02-11T09:42:25.045 INFO:tasks.ceph:Starting mon daemons in cluster c1... 2021-02-11T09:42:25.045 INFO:tasks.ceph.c1.mon.a:Restarting daemon 2021-02-11T09:42:25.045 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c1 -i a 2021-02-11T09:42:25.089 INFO:tasks.ceph.c1.mon.a:Started 2021-02-11T09:42:25.089 INFO:tasks.ceph:Starting mgr daemons in cluster c1... 2021-02-11T09:42:25.089 INFO:tasks.ceph.c1.mgr.x:Restarting daemon 2021-02-11T09:42:25.089 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c1 -i x 2021-02-11T09:42:25.091 INFO:tasks.ceph.c1.mgr.x:Started 2021-02-11T09:42:25.092 INFO:tasks.ceph:Setting crush tunables to default 2021-02-11T09:42:25.092 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph --cluster c1 osd crush tunables default 2021-02-11T09:42:25.483 INFO:teuthology.orchestra.run.smithi094.stderr:adjusted tunables profile to default 2021-02-11T09:42:25.495 INFO:tasks.ceph:Starting osd daemons in cluster c1... 2021-02-11T09:42:25.495 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2021-02-11T09:42:25.496 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/osd/c1-0/fsid of=/dev/stdout 2021-02-11T09:42:25.503 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2021-02-11T09:42:25.504 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/osd/c1-1/fsid of=/dev/stdout 2021-02-11T09:42:25.559 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2021-02-11T09:42:25.559 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/osd/c1-2/fsid of=/dev/stdout 2021-02-11T09:42:25.615 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph --cluster c1 osd new 5ad1fdb3-bb5e-43f4-92ee-e522fd99ffb6 0 2021-02-11T09:42:25.872 INFO:teuthology.orchestra.run.smithi094.stdout:0 2021-02-11T09:42:25.885 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph --cluster c1 osd new f51bf15f-b1fb-47c2-8586-8a36cfcf8ab6 1 2021-02-11T09:42:26.097 INFO:teuthology.orchestra.run.smithi094.stdout:1 2021-02-11T09:42:26.108 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph --cluster c1 osd new fa9f973f-5860-4200-bbfd-7053478e8a62 2 2021-02-11T09:42:26.310 INFO:teuthology.orchestra.run.smithi094.stdout:2 2021-02-11T09:42:26.330 INFO:tasks.ceph.c1.osd.0:Restarting daemon 2021-02-11T09:42:26.330 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 0 2021-02-11T09:42:26.332 INFO:tasks.ceph.c1.osd.0:Started 2021-02-11T09:42:26.332 INFO:tasks.ceph.c1.osd.1:Restarting daemon 2021-02-11T09:42:26.332 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 1 2021-02-11T09:42:26.334 INFO:tasks.ceph.c1.osd.1:Started 2021-02-11T09:42:26.335 INFO:tasks.ceph.c1.osd.2:Restarting daemon 2021-02-11T09:42:26.335 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 2 2021-02-11T09:42:26.337 INFO:tasks.ceph.c1.osd.2:Started 2021-02-11T09:42:26.337 INFO:tasks.ceph:Waiting for OSDs to come up 2021-02-11T09:42:26.640 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2021-02-11T09:42:26.828 INFO:teuthology.misc.health.smithi094.stdout: 2021-02-11T09:42:26.829 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":5,"fsid":"67a12f0c-a3f1-4f95-b0bb-3bdc01f1c8cf","created":"2021-02-11T09:42:25.208182+0000","modified":"2021-02-11T09:42:26.309821+0000","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"5ad1fdb3-bb5e-43f4-92ee-e522fd99ffb6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f51bf15f-b1fb-47c2-8586-8a36cfcf8ab6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fa9f973f-5860-4200-bbfd-7053478e8a62","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-02-11T09:42:26.839 DEBUG:teuthology.misc:0 of 3 OSDs are up 2021-02-11T09:42:26.937 INFO:tasks.ceph.c1.osd.0.smithi094.stderr:2021-02-11T09:42:26.935+0000 7f7dda006080 -1 Falling back to public interface 2021-02-11T09:42:26.937 INFO:tasks.ceph.c1.osd.1.smithi094.stderr:2021-02-11T09:42:26.935+0000 7fea769cd080 -1 Falling back to public interface 2021-02-11T09:42:26.952 INFO:tasks.ceph.c1.osd.2.smithi094.stderr:2021-02-11T09:42:26.951+0000 7fd42be8f080 -1 Falling back to public interface 2021-02-11T09:42:29.848 INFO:tasks.ceph.c1.osd.1.smithi094.stderr:2021-02-11T09:42:29.847+0000 7fea769cd080 -1 osd.1 0 log_to_monitors {default=true} 2021-02-11T09:42:37.466 INFO:tasks.ceph.c1.osd.2.smithi094.stderr:2021-02-11T09:42:29.919+0000 7fd42be8f080 -1 osd.2 0 log_to_monitors {default=true} 2021-02-11T09:42:37.466 INFO:tasks.ceph.c1.osd.2.smithi094.stderr:2021-02-11T09:42:31.927+0000 7fd423358700 -1 osd.2 0 waiting for initial osdmap 2021-02-11T09:42:37.466 INFO:tasks.ceph.c1.osd.0.smithi094.stderr:2021-02-11T09:42:29.919+0000 7f7dda006080 -1 osd.0 0 log_to_monitors {default=true} 2021-02-11T09:42:37.466 INFO:tasks.ceph.c1.osd.0.smithi094.stderr:2021-02-11T09:42:31.927+0000 7f7dd14cf700 -1 osd.0 0 waiting for initial osdmap 2021-02-11T09:42:37.467 INFO:tasks.ceph.c1.osd.1.smithi094.stderr:2021-02-11T09:42:30.919+0000 7fea6de96700 -1 osd.1 0 waiting for initial osdmap 2021-02-11T09:42:37.767 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2021-02-11T09:42:38.060 INFO:teuthology.misc.health.smithi094.stdout: 2021-02-11T09:42:38.061 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":12,"fsid":"67a12f0c-a3f1-4f95-b0bb-3bdc01f1c8cf","created":"2021-02-11T09:42:25.208182+0000","modified":"2021-02-11T09:42:35.942750+0000","last_up_change":"2021-02-11T09:42:32.923194+0000","last_in_change":"2021-02-11T09:42:32.923194+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-02-11T09:42:32.937534+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"5ad1fdb3-bb5e-43f4-92ee-e522fd99ffb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":14390},{"type":"v1","addr":"172.21.15.94:6801","nonce":14390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":14390},{"type":"v1","addr":"172.21.15.94:6803","nonce":14390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":14390},{"type":"v1","addr":"172.21.15.94:6811","nonce":14390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6805","nonce":14390},{"type":"v1","addr":"172.21.15.94:6807","nonce":14390}]},"public_addr":"172.21.15.94:6801/14390","cluster_addr":"172.21.15.94:6803/14390","heartbeat_back_addr":"172.21.15.94:6811/14390","heartbeat_front_addr":"172.21.15.94:6807/14390","state":["exists","up"]},{"osd":1,"uuid":"f51bf15f-b1fb-47c2-8586-8a36cfcf8ab6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":14389},{"type":"v1","addr":"172.21.15.94:6806","nonce":14389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":14389},{"type":"v1","addr":"172.21.15.94:6810","nonce":14389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":14389},{"type":"v1","addr":"172.21.15.94:6815","nonce":14389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":14389},{"type":"v1","addr":"172.21.15.94:6813","nonce":14389}]},"public_addr":"172.21.15.94:6806/14389","cluster_addr":"172.21.15.94:6810/14389","heartbeat_back_addr":"172.21.15.94:6815/14389","heartbeat_front_addr":"172.21.15.94:6813/14389","state":["exists","up"]},{"osd":2,"uuid":"fa9f973f-5860-4200-bbfd-7053478e8a62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":14391},{"type":"v1","addr":"172.21.15.94:6817","nonce":14391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":14391},{"type":"v1","addr":"172.21.15.94:6819","nonce":14391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":14391},{"type":"v1","addr":"172.21.15.94:6823","nonce":14391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":14391},{"type":"v1","addr":"172.21.15.94:6821","nonce":14391}]},"public_addr":"172.21.15.94:6817/14391","cluster_addr":"172.21.15.94:6819/14391","heartbeat_back_addr":"172.21.15.94:6823/14391","heartbeat_front_addr":"172.21.15.94:6821/14391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T09:42:30.925557+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T09:42:30.834864+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T09:42:30.923398+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-02-11T09:42:38.074 DEBUG:teuthology.misc:3 of 3 OSDs are up 2021-02-11T09:42:38.074 INFO:tasks.ceph:Creating RBD pool 2021-02-11T09:42:38.074 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph --cluster c1 osd pool create rbd 8 2021-02-11T09:42:39.063 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'rbd' created 2021-02-11T09:42:39.076 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph --cluster c1 osd pool application enable rbd rbd --yes-i-really-mean-it 2021-02-11T09:42:40.074 INFO:teuthology.orchestra.run.smithi094.stderr:enabled application 'rbd' on pool 'rbd' 2021-02-11T09:42:40.088 INFO:tasks.ceph:Starting mds daemons in cluster c1... 2021-02-11T09:42:40.088 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2021-02-11T09:42:40.089 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2021-02-11T09:42:40.386 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-02-11T09:42:40.386 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":14,"fsid":"67a12f0c-a3f1-4f95-b0bb-3bdc01f1c8cf","created":"2021-02-11T09:42:25.208182+0000","modified":"2021-02-11T09:42:40.056239+0000","last_up_change":"2021-02-11T09:42:32.923194+0000","last_in_change":"2021-02-11T09:42:32.923194+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-02-11T09:42:32.937534+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-02-11T09:42:38.378396+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"5ad1fdb3-bb5e-43f4-92ee-e522fd99ffb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":14390},{"type":"v1","addr":"172.21.15.94:6801","nonce":14390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":14390},{"type":"v1","addr":"172.21.15.94:6803","nonce":14390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":14390},{"type":"v1","addr":"172.21.15.94:6811","nonce":14390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6805","nonce":14390},{"type":"v1","addr":"172.21.15.94:6807","nonce":14390}]},"public_addr":"172.21.15.94:6801/14390","cluster_addr":"172.21.15.94:6803/14390","heartbeat_back_addr":"172.21.15.94:6811/14390","heartbeat_front_addr":"172.21.15.94:6807/14390","state":["exists","up"]},{"osd":1,"uuid":"f51bf15f-b1fb-47c2-8586-8a36cfcf8ab6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":14389},{"type":"v1","addr":"172.21.15.94:6806","nonce":14389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":14389},{"type":"v1","addr":"172.21.15.94:6810","nonce":14389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":14389},{"type":"v1","addr":"172.21.15.94:6815","nonce":14389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":14389},{"type":"v1","addr":"172.21.15.94:6813","nonce":14389}]},"public_addr":"172.21.15.94:6806/14389","cluster_addr":"172.21.15.94:6810/14389","heartbeat_back_addr":"172.21.15.94:6815/14389","heartbeat_front_addr":"172.21.15.94:6813/14389","state":["exists","up"]},{"osd":2,"uuid":"fa9f973f-5860-4200-bbfd-7053478e8a62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":14391},{"type":"v1","addr":"172.21.15.94:6817","nonce":14391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":14391},{"type":"v1","addr":"172.21.15.94:6819","nonce":14391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":14391},{"type":"v1","addr":"172.21.15.94:6823","nonce":14391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":14391},{"type":"v1","addr":"172.21.15.94:6821","nonce":14391}]},"public_addr":"172.21.15.94:6817/14391","cluster_addr":"172.21.15.94:6819/14391","heartbeat_back_addr":"172.21.15.94:6823/14391","heartbeat_front_addr":"172.21.15.94:6821/14391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T09:42:30.925557+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T09:42:30.834864+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T09:42:30.923398+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-02-11T09:42:40.404 INFO:tasks.ceph.ceph_manager.c1:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2021-02-11T09:42:32.937534+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '12', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2021-02-11T09:42:38.378396+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '14', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-11T09:42:40.405 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd pool get device_health_metrics pg_num 2021-02-11T09:42:40.703 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 1 2021-02-11T09:42:40.718 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd pool get rbd pg_num 2021-02-11T09:42:41.008 INFO:teuthology.orchestra.run.smithi094.stdout:pg_num: 8 2021-02-11T09:42:41.022 INFO:tasks.ceph:Waiting until c1 daemons up and pgs clean... 2021-02-11T09:42:41.023 INFO:tasks.ceph.ceph_manager.c1:waiting for mgr available 2021-02-11T09:42:41.023 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 mgr dump --format=json 2021-02-11T09:42:41.360 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-02-11T09:42:41.374 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":4,"active_gid":4099,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":14206},{"type":"v1","addr":"172.21.15.94:6825","nonce":14206}]},"active_addr":"172.21.15.94:6825/14206","active_change":"2021-02-11T09:42:29.987878+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading remoto library:No module named 'remoto'","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"haproxy","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.94:0","nonce":2626058073}]},{"addrvec":[{"type":"v2","addr":"172.21.15.94:0","nonce":3437343448}]}]}} 2021-02-11T09:42:41.376 INFO:tasks.ceph.ceph_manager.c1:mgr available! 2021-02-11T09:42:41.376 INFO:tasks.ceph.ceph_manager.c1:waiting for all up 2021-02-11T09:42:41.376 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2021-02-11T09:42:41.667 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-02-11T09:42:41.667 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":15,"fsid":"67a12f0c-a3f1-4f95-b0bb-3bdc01f1c8cf","created":"2021-02-11T09:42:25.208182+0000","modified":"2021-02-11T09:42:41.061576+0000","last_up_change":"2021-02-11T09:42:32.923194+0000","last_in_change":"2021-02-11T09:42:32.923194+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-02-11T09:42:32.937534+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-02-11T09:42:38.378396+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"5ad1fdb3-bb5e-43f4-92ee-e522fd99ffb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":14390},{"type":"v1","addr":"172.21.15.94:6801","nonce":14390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":14390},{"type":"v1","addr":"172.21.15.94:6803","nonce":14390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":14390},{"type":"v1","addr":"172.21.15.94:6811","nonce":14390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6805","nonce":14390},{"type":"v1","addr":"172.21.15.94:6807","nonce":14390}]},"public_addr":"172.21.15.94:6801/14390","cluster_addr":"172.21.15.94:6803/14390","heartbeat_back_addr":"172.21.15.94:6811/14390","heartbeat_front_addr":"172.21.15.94:6807/14390","state":["exists","up"]},{"osd":1,"uuid":"f51bf15f-b1fb-47c2-8586-8a36cfcf8ab6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":14389},{"type":"v1","addr":"172.21.15.94:6806","nonce":14389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":14389},{"type":"v1","addr":"172.21.15.94:6810","nonce":14389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":14389},{"type":"v1","addr":"172.21.15.94:6815","nonce":14389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":14389},{"type":"v1","addr":"172.21.15.94:6813","nonce":14389}]},"public_addr":"172.21.15.94:6806/14389","cluster_addr":"172.21.15.94:6810/14389","heartbeat_back_addr":"172.21.15.94:6815/14389","heartbeat_front_addr":"172.21.15.94:6813/14389","state":["exists","up"]},{"osd":2,"uuid":"fa9f973f-5860-4200-bbfd-7053478e8a62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":14391},{"type":"v1","addr":"172.21.15.94:6817","nonce":14391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":14391},{"type":"v1","addr":"172.21.15.94:6819","nonce":14391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":14391},{"type":"v1","addr":"172.21.15.94:6823","nonce":14391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":14391},{"type":"v1","addr":"172.21.15.94:6821","nonce":14391}]},"public_addr":"172.21.15.94:6817/14391","cluster_addr":"172.21.15.94:6819/14391","heartbeat_back_addr":"172.21.15.94:6823/14391","heartbeat_front_addr":"172.21.15.94:6821/14391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T09:42:30.925557+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T09:42:30.834864+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T09:42:30.923398+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-02-11T09:42:41.682 INFO:tasks.ceph.ceph_manager.c1:all up! 2021-02-11T09:42:41.682 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2021-02-11T09:42:41.978 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-02-11T09:42:41.978 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":15,"fsid":"67a12f0c-a3f1-4f95-b0bb-3bdc01f1c8cf","created":"2021-02-11T09:42:25.208182+0000","modified":"2021-02-11T09:42:41.061576+0000","last_up_change":"2021-02-11T09:42:32.923194+0000","last_in_change":"2021-02-11T09:42:32.923194+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-02-11T09:42:32.937534+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-02-11T09:42:38.378396+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"5ad1fdb3-bb5e-43f4-92ee-e522fd99ffb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":14390},{"type":"v1","addr":"172.21.15.94:6801","nonce":14390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":14390},{"type":"v1","addr":"172.21.15.94:6803","nonce":14390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":14390},{"type":"v1","addr":"172.21.15.94:6811","nonce":14390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6805","nonce":14390},{"type":"v1","addr":"172.21.15.94:6807","nonce":14390}]},"public_addr":"172.21.15.94:6801/14390","cluster_addr":"172.21.15.94:6803/14390","heartbeat_back_addr":"172.21.15.94:6811/14390","heartbeat_front_addr":"172.21.15.94:6807/14390","state":["exists","up"]},{"osd":1,"uuid":"f51bf15f-b1fb-47c2-8586-8a36cfcf8ab6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":14389},{"type":"v1","addr":"172.21.15.94:6806","nonce":14389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":14389},{"type":"v1","addr":"172.21.15.94:6810","nonce":14389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":14389},{"type":"v1","addr":"172.21.15.94:6815","nonce":14389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":14389},{"type":"v1","addr":"172.21.15.94:6813","nonce":14389}]},"public_addr":"172.21.15.94:6806/14389","cluster_addr":"172.21.15.94:6810/14389","heartbeat_back_addr":"172.21.15.94:6815/14389","heartbeat_front_addr":"172.21.15.94:6813/14389","state":["exists","up"]},{"osd":2,"uuid":"fa9f973f-5860-4200-bbfd-7053478e8a62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":14391},{"type":"v1","addr":"172.21.15.94:6817","nonce":14391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":14391},{"type":"v1","addr":"172.21.15.94:6819","nonce":14391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":14391},{"type":"v1","addr":"172.21.15.94:6823","nonce":14391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":14391},{"type":"v1","addr":"172.21.15.94:6821","nonce":14391}]},"public_addr":"172.21.15.94:6817/14391","cluster_addr":"172.21.15.94:6819/14391","heartbeat_back_addr":"172.21.15.94:6823/14391","heartbeat_front_addr":"172.21.15.94:6821/14391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T09:42:30.925557+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T09:42:30.834864+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T09:42:30.923398+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-02-11T09:42:41.998 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.0 flush_pg_stats 2021-02-11T09:42:42.165 INFO:teuthology.orchestra.run.smithi094.stdout:38654705667 2021-02-11T09:42:42.165 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.1 flush_pg_stats 2021-02-11T09:42:42.334 INFO:teuthology.orchestra.run.smithi094.stdout:34359738373 2021-02-11T09:42:42.334 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.2 flush_pg_stats 2021-02-11T09:42:42.502 INFO:teuthology.orchestra.run.smithi094.stdout:38654705667 2021-02-11T09:42:42.502 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2021-02-11T09:42:42.802 INFO:teuthology.orchestra.run.smithi094.stdout:38654705666 2021-02-11T09:42:42.824 INFO:tasks.ceph.ceph_manager.c1:need seq 38654705667 got 38654705666 for osd.0 2021-02-11T09:42:43.825 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2021-02-11T09:42:44.136 INFO:teuthology.orchestra.run.smithi094.stdout:38654705668 2021-02-11T09:42:44.150 INFO:tasks.ceph.ceph_manager.c1:need seq 38654705667 got 38654705668 for osd.0 2021-02-11T09:42:44.150 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2021-02-11T09:42:44.441 INFO:teuthology.orchestra.run.smithi094.stdout:34359738373 2021-02-11T09:42:44.456 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738373 got 34359738373 for osd.1 2021-02-11T09:42:44.456 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2021-02-11T09:42:44.748 INFO:teuthology.orchestra.run.smithi094.stdout:38654705668 2021-02-11T09:42:44.763 INFO:tasks.ceph.ceph_manager.c1:need seq 38654705667 got 38654705668 for osd.2 2021-02-11T09:42:44.763 INFO:tasks.ceph.ceph_manager.c1:waiting for clean 2021-02-11T09:42:44.763 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2021-02-11T09:42:45.053 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-02-11T09:42:45.056 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2021-02-11T09:42:45.067 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2021-02-11T09:42:43.997911+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":14800,"kb_used_data":376,"kb_used_omap":0,"kb_used_meta":14400,"kb_avail":314558000,"statfs":{"total":322122547200,"available":322107392000,"internally_reserved":0,"allocated":385024,"data_stored":109032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":14745600},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"8","reported_epoch":"14","state":"active+clean","last_fresh":"2021-02-11T09:42:40.076205+0000","last_change":"2021-02-11T09:42:40.076205+0000","last_active":"2021-02-11T09:42:40.076205+0000","last_peered":"2021-02-11T09:42:40.076205+0000","last_clean":"2021-02-11T09:42:40.076205+0000","last_became_active":"2021-02-11T09:42:40.074939+0000","last_became_peered":"2021-02-11T09:42:40.074939+0000","last_unstale":"2021-02-11T09:42:40.076205+0000","last_undegraded":"2021-02-11T09:42:40.076205+0000","last_fullsized":"2021-02-11T09:42:40.076205+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T09:42:39.049369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T09:42:39.049369+0000","last_clean_scrub_stamp":"2021-02-11T09:42:39.049369+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"8","reported_epoch":"14","state":"active+clean","last_fresh":"2021-02-11T09:42:40.077155+0000","last_change":"2021-02-11T09:42:40.077155+0000","last_active":"2021-02-11T09:42:40.077155+0000","last_peered":"2021-02-11T09:42:40.077155+0000","last_clean":"2021-02-11T09:42:40.077155+0000","last_became_active":"2021-02-11T09:42:40.076809+0000","last_became_peered":"2021-02-11T09:42:40.076809+0000","last_unstale":"2021-02-11T09:42:40.077155+0000","last_undegraded":"2021-02-11T09:42:40.077155+0000","last_fullsized":"2021-02-11T09:42:40.077155+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T09:42:39.049369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T09:42:39.049369+0000","last_clean_scrub_stamp":"2021-02-11T09:42:39.049369+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"8","reported_epoch":"14","state":"active+clean","last_fresh":"2021-02-11T09:42:40.074758+0000","last_change":"2021-02-11T09:42:40.074758+0000","last_active":"2021-02-11T09:42:40.074758+0000","last_peered":"2021-02-11T09:42:40.074758+0000","last_clean":"2021-02-11T09:42:40.074758+0000","last_became_active":"2021-02-11T09:42:40.073858+0000","last_became_peered":"2021-02-11T09:42:40.073858+0000","last_unstale":"2021-02-11T09:42:40.074758+0000","last_undegraded":"2021-02-11T09:42:40.074758+0000","last_fullsized":"2021-02-11T09:42:40.074758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T09:42:39.049369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T09:42:39.049369+0000","last_clean_scrub_stamp":"2021-02-11T09:42:39.049369+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2021-02-11T09:42:40.079419+0000","last_change":"2021-02-11T09:42:40.078913+0000","last_active":"2021-02-11T09:42:40.079419+0000","last_peered":"2021-02-11T09:42:40.079419+0000","last_clean":"2021-02-11T09:42:40.079419+0000","last_became_active":"2021-02-11T09:42:40.078487+0000","last_became_peered":"2021-02-11T09:42:40.078487+0000","last_unstale":"2021-02-11T09:42:40.079419+0000","last_undegraded":"2021-02-11T09:42:40.079419+0000","last_fullsized":"2021-02-11T09:42:40.079419+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T09:42:39.049369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T09:42:39.049369+0000","last_clean_scrub_stamp":"2021-02-11T09:42:39.049369+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"11","reported_epoch":"13","state":"active+clean","last_fresh":"2021-02-11T09:42:39.058427+0000","last_change":"2021-02-11T09:42:34.957936+0000","last_active":"2021-02-11T09:42:39.058427+0000","last_peered":"2021-02-11T09:42:39.058427+0000","last_clean":"2021-02-11T09:42:39.058427+0000","last_became_active":"2021-02-11T09:42:34.957490+0000","last_became_peered":"2021-02-11T09:42:34.957490+0000","last_unstale":"2021-02-11T09:42:39.058427+0000","last_undegraded":"2021-02-11T09:42:39.058427+0000","last_fullsized":"2021-02-11T09:42:39.058427+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T09:42:33.931569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T09:42:33.931569+0000","last_clean_scrub_stamp":"2021-02-11T09:42:33.931569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"8","reported_epoch":"14","state":"active+clean","last_fresh":"2021-02-11T09:42:40.075218+0000","last_change":"2021-02-11T09:42:40.075218+0000","last_active":"2021-02-11T09:42:40.075218+0000","last_peered":"2021-02-11T09:42:40.075218+0000","last_clean":"2021-02-11T09:42:40.075218+0000","last_became_active":"2021-02-11T09:42:40.074786+0000","last_became_peered":"2021-02-11T09:42:40.074786+0000","last_unstale":"2021-02-11T09:42:40.075218+0000","last_undegraded":"2021-02-11T09:42:40.075218+0000","last_fullsized":"2021-02-11T09:42:40.075218+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T09:42:39.049369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T09:42:39.049369+0000","last_clean_scrub_stamp":"2021-02-11T09:42:39.049369+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2021-02-11T09:42:40.078050+0000","last_change":"2021-02-11T09:42:40.077671+0000","last_active":"2021-02-11T09:42:40.078050+0000","last_peered":"2021-02-11T09:42:40.078050+0000","last_clean":"2021-02-11T09:42:40.078050+0000","last_became_active":"2021-02-11T09:42:40.077322+0000","last_became_peered":"2021-02-11T09:42:40.077322+0000","last_unstale":"2021-02-11T09:42:40.078050+0000","last_undegraded":"2021-02-11T09:42:40.078050+0000","last_fullsized":"2021-02-11T09:42:40.078050+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T09:42:39.049369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T09:42:39.049369+0000","last_clean_scrub_stamp":"2021-02-11T09:42:39.049369+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2021-02-11T09:42:40.074389+0000","last_change":"2021-02-11T09:42:40.073368+0000","last_active":"2021-02-11T09:42:40.074389+0000","last_peered":"2021-02-11T09:42:40.074389+0000","last_clean":"2021-02-11T09:42:40.074389+0000","last_became_active":"2021-02-11T09:42:40.073060+0000","last_became_peered":"2021-02-11T09:42:40.073060+0000","last_unstale":"2021-02-11T09:42:40.074389+0000","last_undegraded":"2021-02-11T09:42:40.074389+0000","last_fullsized":"2021-02-11T09:42:40.074389+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T09:42:39.049369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T09:42:39.049369+0000","last_clean_scrub_stamp":"2021-02-11T09:42:39.049369+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"8","reported_epoch":"14","state":"active+clean","last_fresh":"2021-02-11T09:42:40.076567+0000","last_change":"2021-02-11T09:42:40.076567+0000","last_active":"2021-02-11T09:42:40.076567+0000","last_peered":"2021-02-11T09:42:40.076567+0000","last_clean":"2021-02-11T09:42:40.076567+0000","last_became_active":"2021-02-11T09:42:40.075786+0000","last_became_peered":"2021-02-11T09:42:40.075786+0000","last_unstale":"2021-02-11T09:42:40.076567+0000","last_undegraded":"2021-02-11T09:42:40.076567+0000","last_fullsized":"2021-02-11T09:42:40.076567+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T09:42:39.049369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T09:42:39.049369+0000","last_clean_scrub_stamp":"2021-02-11T09:42:39.049369+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705668,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4936,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852664,"statfs":{"total":107374182400,"available":107369127936,"internally_reserved":0,"allocated":131072,"data_stored":37515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3400000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.141}]}]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4928,"kb_used_data":120,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852672,"statfs":{"total":107374182400,"available":107369136128,"internally_reserved":0,"allocated":122880,"data_stored":34002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.413}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.778}]}]},{"osd":0,"up_from":9,"seq":38654705668,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4936,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852664,"statfs":{"total":107374182400,"available":107369127936,"internally_reserved":0,"allocated":131072,"data_stored":37515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.363}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.784}]}]}]}} 2021-02-11T09:42:45.068 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2021-02-11T09:42:45.361 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-02-11T09:42:45.363 INFO:teuthology.orchestra.run.smithi094.stderr:dumped all 2021-02-11T09:42:45.375 INFO:teuthology.orchestra.run.smithi094.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2021-02-11T09:42:43.997911+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":14800,"kb_used_data":376,"kb_used_omap":0,"kb_used_meta":14400,"kb_avail":314558000,"statfs":{"total":322122547200,"available":322107392000,"internally_reserved":0,"allocated":385024,"data_stored":109032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":14745600},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"8","reported_epoch":"14","state":"active+clean","last_fresh":"2021-02-11T09:42:40.076205+0000","last_change":"2021-02-11T09:42:40.076205+0000","last_active":"2021-02-11T09:42:40.076205+0000","last_peered":"2021-02-11T09:42:40.076205+0000","last_clean":"2021-02-11T09:42:40.076205+0000","last_became_active":"2021-02-11T09:42:40.074939+0000","last_became_peered":"2021-02-11T09:42:40.074939+0000","last_unstale":"2021-02-11T09:42:40.076205+0000","last_undegraded":"2021-02-11T09:42:40.076205+0000","last_fullsized":"2021-02-11T09:42:40.076205+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T09:42:39.049369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T09:42:39.049369+0000","last_clean_scrub_stamp":"2021-02-11T09:42:39.049369+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"8","reported_epoch":"14","state":"active+clean","last_fresh":"2021-02-11T09:42:40.077155+0000","last_change":"2021-02-11T09:42:40.077155+0000","last_active":"2021-02-11T09:42:40.077155+0000","last_peered":"2021-02-11T09:42:40.077155+0000","last_clean":"2021-02-11T09:42:40.077155+0000","last_became_active":"2021-02-11T09:42:40.076809+0000","last_became_peered":"2021-02-11T09:42:40.076809+0000","last_unstale":"2021-02-11T09:42:40.077155+0000","last_undegraded":"2021-02-11T09:42:40.077155+0000","last_fullsized":"2021-02-11T09:42:40.077155+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T09:42:39.049369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T09:42:39.049369+0000","last_clean_scrub_stamp":"2021-02-11T09:42:39.049369+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"8","reported_epoch":"14","state":"active+clean","last_fresh":"2021-02-11T09:42:40.074758+0000","last_change":"2021-02-11T09:42:40.074758+0000","last_active":"2021-02-11T09:42:40.074758+0000","last_peered":"2021-02-11T09:42:40.074758+0000","last_clean":"2021-02-11T09:42:40.074758+0000","last_became_active":"2021-02-11T09:42:40.073858+0000","last_became_peered":"2021-02-11T09:42:40.073858+0000","last_unstale":"2021-02-11T09:42:40.074758+0000","last_undegraded":"2021-02-11T09:42:40.074758+0000","last_fullsized":"2021-02-11T09:42:40.074758+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T09:42:39.049369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T09:42:39.049369+0000","last_clean_scrub_stamp":"2021-02-11T09:42:39.049369+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2021-02-11T09:42:40.079419+0000","last_change":"2021-02-11T09:42:40.078913+0000","last_active":"2021-02-11T09:42:40.079419+0000","last_peered":"2021-02-11T09:42:40.079419+0000","last_clean":"2021-02-11T09:42:40.079419+0000","last_became_active":"2021-02-11T09:42:40.078487+0000","last_became_peered":"2021-02-11T09:42:40.078487+0000","last_unstale":"2021-02-11T09:42:40.079419+0000","last_undegraded":"2021-02-11T09:42:40.079419+0000","last_fullsized":"2021-02-11T09:42:40.079419+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T09:42:39.049369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T09:42:39.049369+0000","last_clean_scrub_stamp":"2021-02-11T09:42:39.049369+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"11","reported_epoch":"13","state":"active+clean","last_fresh":"2021-02-11T09:42:39.058427+0000","last_change":"2021-02-11T09:42:34.957936+0000","last_active":"2021-02-11T09:42:39.058427+0000","last_peered":"2021-02-11T09:42:39.058427+0000","last_clean":"2021-02-11T09:42:39.058427+0000","last_became_active":"2021-02-11T09:42:34.957490+0000","last_became_peered":"2021-02-11T09:42:34.957490+0000","last_unstale":"2021-02-11T09:42:39.058427+0000","last_undegraded":"2021-02-11T09:42:39.058427+0000","last_fullsized":"2021-02-11T09:42:39.058427+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T09:42:33.931569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T09:42:33.931569+0000","last_clean_scrub_stamp":"2021-02-11T09:42:33.931569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"8","reported_epoch":"14","state":"active+clean","last_fresh":"2021-02-11T09:42:40.075218+0000","last_change":"2021-02-11T09:42:40.075218+0000","last_active":"2021-02-11T09:42:40.075218+0000","last_peered":"2021-02-11T09:42:40.075218+0000","last_clean":"2021-02-11T09:42:40.075218+0000","last_became_active":"2021-02-11T09:42:40.074786+0000","last_became_peered":"2021-02-11T09:42:40.074786+0000","last_unstale":"2021-02-11T09:42:40.075218+0000","last_undegraded":"2021-02-11T09:42:40.075218+0000","last_fullsized":"2021-02-11T09:42:40.075218+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T09:42:39.049369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T09:42:39.049369+0000","last_clean_scrub_stamp":"2021-02-11T09:42:39.049369+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2021-02-11T09:42:40.078050+0000","last_change":"2021-02-11T09:42:40.077671+0000","last_active":"2021-02-11T09:42:40.078050+0000","last_peered":"2021-02-11T09:42:40.078050+0000","last_clean":"2021-02-11T09:42:40.078050+0000","last_became_active":"2021-02-11T09:42:40.077322+0000","last_became_peered":"2021-02-11T09:42:40.077322+0000","last_unstale":"2021-02-11T09:42:40.078050+0000","last_undegraded":"2021-02-11T09:42:40.078050+0000","last_fullsized":"2021-02-11T09:42:40.078050+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T09:42:39.049369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T09:42:39.049369+0000","last_clean_scrub_stamp":"2021-02-11T09:42:39.049369+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2021-02-11T09:42:40.074389+0000","last_change":"2021-02-11T09:42:40.073368+0000","last_active":"2021-02-11T09:42:40.074389+0000","last_peered":"2021-02-11T09:42:40.074389+0000","last_clean":"2021-02-11T09:42:40.074389+0000","last_became_active":"2021-02-11T09:42:40.073060+0000","last_became_peered":"2021-02-11T09:42:40.073060+0000","last_unstale":"2021-02-11T09:42:40.074389+0000","last_undegraded":"2021-02-11T09:42:40.074389+0000","last_fullsized":"2021-02-11T09:42:40.074389+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T09:42:39.049369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T09:42:39.049369+0000","last_clean_scrub_stamp":"2021-02-11T09:42:39.049369+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"8","reported_epoch":"14","state":"active+clean","last_fresh":"2021-02-11T09:42:40.076567+0000","last_change":"2021-02-11T09:42:40.076567+0000","last_active":"2021-02-11T09:42:40.076567+0000","last_peered":"2021-02-11T09:42:40.076567+0000","last_clean":"2021-02-11T09:42:40.076567+0000","last_became_active":"2021-02-11T09:42:40.075786+0000","last_became_peered":"2021-02-11T09:42:40.075786+0000","last_unstale":"2021-02-11T09:42:40.076567+0000","last_undegraded":"2021-02-11T09:42:40.076567+0000","last_fullsized":"2021-02-11T09:42:40.076567+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T09:42:39.049369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T09:42:39.049369+0000","last_clean_scrub_stamp":"2021-02-11T09:42:39.049369+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705668,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4936,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852664,"statfs":{"total":107374182400,"available":107369127936,"internally_reserved":0,"allocated":131072,"data_stored":37515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3400000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.141}]}]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4928,"kb_used_data":120,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852672,"statfs":{"total":107374182400,"available":107369136128,"internally_reserved":0,"allocated":122880,"data_stored":34002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.413}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.778}]}]},{"osd":0,"up_from":9,"seq":38654705668,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4936,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852664,"statfs":{"total":107374182400,"available":107369127936,"internally_reserved":0,"allocated":131072,"data_stored":37515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.363}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.784}]}]}]}} 2021-02-11T09:42:45.376 INFO:tasks.ceph.ceph_manager.c1:clean! 2021-02-11T09:42:45.376 INFO:tasks.ceph:Waiting until ceph cluster c1 is healthy... 2021-02-11T09:42:45.376 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy 2021-02-11T09:42:45.376 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 health --format=json 2021-02-11T09:42:45.727 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-02-11T09:42:45.740 INFO:teuthology.orchestra.run.smithi094.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2021-02-11T09:42:45.740 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy done 2021-02-11T09:42:45.740 INFO:teuthology.run_tasks:Running task ceph... 2021-02-11T09:42:45.749 INFO:tasks.ceph:Creating ceph cluster c2... 2021-02-11T09:42:45.749 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'cpu_profile': set(), 'cluster': 'c2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2021-02-11T09:42:45.749 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/aemerson-2021-02-11_09:24:13-rgw-wip-blurpletastic-distro-basic-smithi/5874501', 'branch': 'wip-blurpletastic', 'description': 'rgw/multisite/{clusters frontend/beast omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind}', 'email': 'aemerson@redhat.com', 'first_in_suite': False, 'job_id': '5874501', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'aemerson-2021-02-11_09:24:13-rgw-wip-blurpletastic-distro-basic-smithi', 'nuke-on-error': True, 'os_type': 'ubuntu', 'overrides': {'admin_socket': {'branch': 'wip-blurpletastic'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '5e5bb33c614118d912ca83c2e5f2be67b3acc04f', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': '5e5bb33c614118d912ca83c2e5f2be67b3acc04f'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}, {'endpoints': ['c2.client.1'], 'is_pubsub': True, 'name': 'test-zone4'}]}]}, 'workunit': {'branch': 'wip-blurpletastic', 'sha1': '5e5bb33c614118d912ca83c2e5f2be67b3acc04f'}}, 'owner': 'scheduled_aemerson@teuthology', 'priority': 70, 'repo': 'git://git.ceph.com/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'sha1': '5e5bb33c614118d912ca83c2e5f2be67b3acc04f', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'wip-blurpletastic', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_5e5bb33c614118d912ca83c2e5f2be67b3acc04f/qa', 'suite_relpath': 'qa', 'suite_repo': 'git://git.ceph.com/ceph-ci.git', 'suite_sha1': '5e5bb33c614118d912ca83c2e5f2be67b3acc04f', 'targets': {'smithi094.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDhsR+Lt7mvR59HsmPpNWxJNim8Xttba7vcp+ScC6u0Pswb6S42rmwG38WbZuU/7ep6pP2x3nzFqADW1Wnw7gA0s7j0f97KmM/1gwSTmn7wLM3vmH9YflZd0u92ejdlWXqzSlZGVQs6OeTXosJev3vWCuWPWzrqKD9PcoohB43I5OcEHwXdYw6yfox2Ev1xwUHJLfFoqPhuuTXsREJrjwfc7wnu0R/x2Uq6TVNZZGZAi4ZUjD0dDShWWpBVBBLR9CY/ld75UTGV43izFmc0s3LqjcZXoTL7Dy3PyhObcx5YlnexpyOFnC9WjKvvd8YM55slmtssLnOCuSO7f2gBX4XH', 'smithi098.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFrxDgXyNFGHyNBYsLuHdnF6Guzsz8LugH6RDGsPAxVb7lwHj53vn1Bo+4ZTv2cbfaDSCx15SSePh5B5Bxej/1IbcypypuZmr928LW50imJaz1UqRratL3KhgE8fC6+dkFKflLidlrAGSq3WPgRGsNq1Y+wMnGK4NC9+LUHRDlGLfPOBTZcYo/SOVg/dVnubnyp+T043bk/jvhjtHNprVq1WMV0i/CyvW46Sc9NaUetAF24NIx3lxL0jxLEcDqVV4DpcpyXK2zWpNwtNxUq1erqDESFP5p/m19cA6YiJsRqwyyYjZVWdb0ZpBTK9RZAB2iO+92elpGyJchAJ4jHaSN'}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '5e5bb33c614118d912ca83c2e5f2be67b3acc04f', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '5e5bb33c614118d912ca83c2e5f2be67b3acc04f', 'wait-for-scrub': False}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology_branch': 'master', 'timestamp': '2021-02-11_09:24:13', 'tube': 'smithi', 'user': 'aemerson', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.29888'} 2021-02-11T09:42:45.750 DEBUG:teuthology.orchestra.run.smithi094:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2021-02-11T09:42:45.752 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2021-02-11T09:42:45.758 DEBUG:teuthology.orchestra.run.smithi094:> sudo install -d -m0777 -- /var/run/ceph 2021-02-11T09:42:45.800 DEBUG:teuthology.orchestra.run.smithi098:> sudo install -d -m0777 -- /var/run/ceph 2021-02-11T09:42:45.813 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2021-02-11T09:42:45.813 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/scratch_devs of=/dev/stdout 2021-02-11T09:42:45.860 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-02-11T09:42:45.860 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-02-11T09:42:45.909 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2021-02-11T09:42:45.909 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-11T09:42:45.909 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 6h/6d Inode: 341 Links: 1 2021-02-11T09:42:45.909 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-11T09:42:45.910 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2021-02-11 09:39:38.098622996 +0000 2021-02-11T09:42:45.910 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2021-02-11 09:39:37.986678996 +0000 2021-02-11T09:42:45.910 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2021-02-11 09:39:37.986678996 +0000 2021-02-11T09:42:45.910 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: - 2021-02-11T09:42:45.918 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2021-02-11T09:42:45.919 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2021-02-11T09:42:45.919 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000343405 s, 1.5 MB/s 2021-02-11T09:42:45.922 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-02-11T09:42:45.969 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2021-02-11T09:42:45.969 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-11T09:42:45.969 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 6h/6d Inode: 348 Links: 1 2021-02-11T09:42:45.969 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-11T09:42:45.969 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2021-02-11 09:39:38.242550998 +0000 2021-02-11T09:42:45.970 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2021-02-11 09:39:38.238552998 +0000 2021-02-11T09:42:45.970 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2021-02-11 09:39:38.238552998 +0000 2021-02-11T09:42:45.970 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: - 2021-02-11T09:42:45.979 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2021-02-11T09:42:45.979 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2021-02-11T09:42:45.979 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000350825 s, 1.5 MB/s 2021-02-11T09:42:45.984 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-02-11T09:42:46.033 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2021-02-11T09:42:46.033 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-11T09:42:46.033 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 6h/6d Inode: 354 Links: 1 2021-02-11T09:42:46.033 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-11T09:42:46.033 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2021-02-11 09:39:38.450446999 +0000 2021-02-11T09:42:46.033 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2021-02-11 09:39:38.446448999 +0000 2021-02-11T09:42:46.034 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2021-02-11 09:39:38.446448999 +0000 2021-02-11T09:42:46.034 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: - 2021-02-11T09:42:46.042 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2021-02-11T09:42:46.042 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2021-02-11T09:42:46.043 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000322052 s, 1.6 MB/s 2021-02-11T09:42:46.046 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-02-11T09:42:46.097 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2021-02-11T09:42:46.097 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-11T09:42:46.097 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 6h/6d Inode: 360 Links: 1 2021-02-11T09:42:46.097 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-11T09:42:46.097 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2021-02-11 09:39:38.670337000 +0000 2021-02-11T09:42:46.097 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2021-02-11 09:39:38.666338999 +0000 2021-02-11T09:42:46.098 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2021-02-11 09:39:38.666338999 +0000 2021-02-11T09:42:46.098 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: - 2021-02-11T09:42:46.106 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2021-02-11T09:42:46.106 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2021-02-11T09:42:46.107 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000389604 s, 1.3 MB/s 2021-02-11T09:42:46.111 INFO:tasks.ceph:osd dev map: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2021-02-11T09:42:46.111 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi098.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2021-02-11T09:42:46.111 INFO:tasks.ceph:Generating config... 2021-02-11T09:42:46.119 INFO:tasks.ceph:[client] debug rgw = 20 2021-02-11T09:42:46.119 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2021-02-11T09:42:46.120 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2021-02-11T09:42:46.120 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-02-11T09:42:46.120 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2021-02-11T09:42:46.120 INFO:tasks.ceph:[client] rgw data log num shards = 4 2021-02-11T09:42:46.120 INFO:tasks.ceph:[client] rgw md log max shards = 4 2021-02-11T09:42:46.120 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2021-02-11T09:42:46.120 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2021-02-11T09:42:46.120 INFO:tasks.ceph:[client] setgroup = ceph 2021-02-11T09:42:46.120 INFO:tasks.ceph:[client] setuser = ceph 2021-02-11T09:42:46.120 INFO:tasks.ceph:[global] osd heartbeat grace = 40 2021-02-11T09:42:46.120 INFO:tasks.ceph:[mgr] debug mgr = 20 2021-02-11T09:42:46.120 INFO:tasks.ceph:[mgr] debug ms = 1 2021-02-11T09:42:46.120 INFO:tasks.ceph:[mon] debug mon = 20 2021-02-11T09:42:46.120 INFO:tasks.ceph:[mon] debug ms = 1 2021-02-11T09:42:46.121 INFO:tasks.ceph:[mon] debug paxos = 20 2021-02-11T09:42:46.121 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2021-02-11T09:42:46.121 INFO:tasks.ceph:[osd] debug filestore = 20 2021-02-11T09:42:46.121 INFO:tasks.ceph:[osd] debug journal = 20 2021-02-11T09:42:46.121 INFO:tasks.ceph:[osd] debug ms = 20 2021-02-11T09:42:46.121 INFO:tasks.ceph:[osd] debug osd = 25 2021-02-11T09:42:46.121 INFO:tasks.ceph:[osd] osd fast shutdown = False 2021-02-11T09:42:46.121 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2021-02-11T09:42:46.121 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2021-02-11T09:42:46.121 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2021-02-11T09:42:46.121 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2021-02-11T09:42:46.121 INFO:tasks.ceph:Setting up c2.mon.a... 2021-02-11T09:42:46.122 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c2.keyring 2021-02-11T09:42:46.186 INFO:teuthology.orchestra.run.smithi098.stdout:creating /etc/ceph/c2.keyring 2021-02-11T09:42:46.188 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c2.keyring 2021-02-11T09:42:46.264 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0644 /etc/ceph/c2.keyring 2021-02-11T09:42:46.319 DEBUG:tasks.ceph:Ceph mon addresses: [('c2.mon.a', '172.21.15.98')] 2021-02-11T09:42:46.319 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.98', 'osd heartbeat grace': 40}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'mon osd crush smoke test': False}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2021-02-11T09:42:46.320 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2021-02-11T09:42:46.320 DEBUG:teuthology.orchestra.run.smithi098:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2021-02-11T09:42:46.368 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.98 --print /home/ubuntu/cephtest/c2.monmap 2021-02-11T09:42:46.440 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:42:46.440 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:42:46.440 INFO:teuthology.orchestra.run.smithi098.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c2.monmap 2021-02-11T09:42:46.440 INFO:teuthology.orchestra.run.smithi098.stdout:monmaptool: generated fsid 0880a54d-a917-41e0-b7d6-67af274484d8 2021-02-11T09:42:46.440 INFO:teuthology.orchestra.run.smithi098.stdout:epoch 0 2021-02-11T09:42:46.440 INFO:teuthology.orchestra.run.smithi098.stdout:fsid 0880a54d-a917-41e0-b7d6-67af274484d8 2021-02-11T09:42:46.441 INFO:teuthology.orchestra.run.smithi098.stdout:last_changed 2021-02-11T09:42:46.444287+0000 2021-02-11T09:42:46.441 INFO:teuthology.orchestra.run.smithi098.stdout:created 2021-02-11T09:42:46.444287+0000 2021-02-11T09:42:46.441 INFO:teuthology.orchestra.run.smithi098.stdout:min_mon_release 0 (unknown) 2021-02-11T09:42:46.441 INFO:teuthology.orchestra.run.smithi098.stdout:election_strategy: 1 2021-02-11T09:42:46.441 INFO:teuthology.orchestra.run.smithi098.stdout:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2021-02-11T09:42:46.441 INFO:teuthology.orchestra.run.smithi098.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c2.monmap (1 monitors) 2021-02-11T09:42:46.442 DEBUG:teuthology.orchestra.run.smithi098:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2021-02-11T09:42:46.488 INFO:tasks.ceph:Writing /etc/ceph/c2.conf for FSID 0880a54d-a917-41e0-b7d6-67af274484d8... 2021-02-11T09:42:46.489 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2021-02-11T09:42:46.491 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2021-02-11T09:42:46.533 INFO:teuthology.orchestra.run.smithi094.stdout:[global] 2021-02-11T09:42:46.533 INFO:teuthology.orchestra.run.smithi094.stdout: chdir = "" 2021-02-11T09:42:46.533 INFO:teuthology.orchestra.run.smithi094.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-02-11T09:42:46.533 INFO:teuthology.orchestra.run.smithi094.stdout: auth supported = cephx 2021-02-11T09:42:46.534 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-02-11T09:42:46.534 INFO:teuthology.orchestra.run.smithi094.stdout: filestore xattr use omap = true 2021-02-11T09:42:46.534 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-02-11T09:42:46.534 INFO:teuthology.orchestra.run.smithi094.stdout: mon clock drift allowed = 1.000 2021-02-11T09:42:46.534 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-02-11T09:42:46.534 INFO:teuthology.orchestra.run.smithi094.stdout: osd crush chooseleaf type = 0 2021-02-11T09:42:46.534 INFO:teuthology.orchestra.run.smithi094.stdout: auth debug = true 2021-02-11T09:42:46.534 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-02-11T09:42:46.534 INFO:teuthology.orchestra.run.smithi094.stdout: ms die on old message = true 2021-02-11T09:42:46.534 INFO:teuthology.orchestra.run.smithi094.stdout: ms die on bug = true 2021-02-11T09:42:46.534 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-02-11T09:42:46.534 INFO:teuthology.orchestra.run.smithi094.stdout: mon max pg per osd = 10000 # >= luminous 2021-02-11T09:42:46.534 INFO:teuthology.orchestra.run.smithi094.stdout: mon pg warn max object skew = 0 2021-02-11T09:42:46.534 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-02-11T09:42:46.535 INFO:teuthology.orchestra.run.smithi094.stdout: # disable pg_autoscaler by default for new pools 2021-02-11T09:42:46.535 INFO:teuthology.orchestra.run.smithi094.stdout: osd_pool_default_pg_autoscale_mode = off 2021-02-11T09:42:46.535 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-02-11T09:42:46.535 INFO:teuthology.orchestra.run.smithi094.stdout: osd pool default size = 2 2021-02-11T09:42:46.535 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-02-11T09:42:46.535 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd allow primary affinity = true 2021-02-11T09:42:46.535 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd allow pg remap = true 2021-02-11T09:42:46.535 INFO:teuthology.orchestra.run.smithi094.stdout: mon warn on legacy crush tunables = false 2021-02-11T09:42:46.535 INFO:teuthology.orchestra.run.smithi094.stdout: mon warn on crush straw calc version zero = false 2021-02-11T09:42:46.535 INFO:teuthology.orchestra.run.smithi094.stdout: mon warn on no sortbitwise = false 2021-02-11T09:42:46.535 INFO:teuthology.orchestra.run.smithi094.stdout: mon warn on osd down out interval zero = false 2021-02-11T09:42:46.535 INFO:teuthology.orchestra.run.smithi094.stdout: mon warn on too few osds = false 2021-02-11T09:42:46.535 INFO:teuthology.orchestra.run.smithi094.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-02-11T09:42:46.535 INFO:teuthology.orchestra.run.smithi094.stdout: mon_warn_on_pool_no_redundancy = false 2021-02-11T09:42:46.536 INFO:teuthology.orchestra.run.smithi094.stdout: mon_allow_pool_size_one = true 2021-02-11T09:42:46.536 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-02-11T09:42:46.536 INFO:teuthology.orchestra.run.smithi094.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-02-11T09:42:46.536 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-02-11T09:42:46.536 INFO:teuthology.orchestra.run.smithi094.stdout: osd default data pool replay window = 5 2021-02-11T09:42:46.536 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-02-11T09:42:46.536 INFO:teuthology.orchestra.run.smithi094.stdout: mon allow pool delete = true 2021-02-11T09:42:46.536 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-02-11T09:42:46.536 INFO:teuthology.orchestra.run.smithi094.stdout: mon cluster log file level = debug 2021-02-11T09:42:46.536 INFO:teuthology.orchestra.run.smithi094.stdout: debug asserts on shutdown = true 2021-02-11T09:42:46.536 INFO:teuthology.orchestra.run.smithi094.stdout: mon health detail to clog = false 2021-02-11T09:42:46.536 INFO:teuthology.orchestra.run.smithi094.stdout: mon host = 172.21.15.98 2021-02-11T09:42:46.536 INFO:teuthology.orchestra.run.smithi094.stdout: osd heartbeat grace = 40 2021-02-11T09:42:46.536 INFO:teuthology.orchestra.run.smithi094.stdout: fsid = 0880a54d-a917-41e0-b7d6-67af274484d8 2021-02-11T09:42:46.537 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-02-11T09:42:46.537 INFO:teuthology.orchestra.run.smithi094.stdout:[osd] 2021-02-11T09:42:46.537 INFO:teuthology.orchestra.run.smithi094.stdout: osd journal size = 100 2021-02-11T09:42:46.537 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-02-11T09:42:46.537 INFO:teuthology.orchestra.run.smithi094.stdout: osd scrub load threshold = 5.0 2021-02-11T09:42:46.537 INFO:teuthology.orchestra.run.smithi094.stdout: osd scrub max interval = 600 2021-02-11T09:42:46.537 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-02-11T09:42:46.537 INFO:teuthology.orchestra.run.smithi094.stdout: osd recover clone overlap = true 2021-02-11T09:42:46.537 INFO:teuthology.orchestra.run.smithi094.stdout: osd recovery max chunk = 1048576 2021-02-11T09:42:46.537 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-02-11T09:42:46.537 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug shutdown = true 2021-02-11T09:42:46.537 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug op order = true 2021-02-11T09:42:46.537 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug verify stray on activate = true 2021-02-11T09:42:46.537 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-02-11T09:42:46.537 INFO:teuthology.orchestra.run.smithi094.stdout: osd open classes on start = true 2021-02-11T09:42:46.538 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug pg log writeout = true 2021-02-11T09:42:46.538 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-02-11T09:42:46.538 INFO:teuthology.orchestra.run.smithi094.stdout: osd deep scrub update digest min age = 30 2021-02-11T09:42:46.538 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-02-11T09:42:46.538 INFO:teuthology.orchestra.run.smithi094.stdout: osd map max advance = 10 2021-02-11T09:42:46.538 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-02-11T09:42:46.538 INFO:teuthology.orchestra.run.smithi094.stdout: journal zero on create = true 2021-02-11T09:42:46.538 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-02-11T09:42:46.538 INFO:teuthology.orchestra.run.smithi094.stdout: filestore ondisk finisher threads = 3 2021-02-11T09:42:46.538 INFO:teuthology.orchestra.run.smithi094.stdout: filestore apply finisher threads = 3 2021-02-11T09:42:46.538 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-02-11T09:42:46.538 INFO:teuthology.orchestra.run.smithi094.stdout: bdev debug aio = true 2021-02-11T09:42:46.538 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug misdirected ops = true 2021-02-11T09:42:46.538 INFO:teuthology.orchestra.run.smithi094.stdout: debug filestore = 20 2021-02-11T09:42:46.538 INFO:teuthology.orchestra.run.smithi094.stdout: debug journal = 20 2021-02-11T09:42:46.539 INFO:teuthology.orchestra.run.smithi094.stdout: debug ms = 20 2021-02-11T09:42:46.539 INFO:teuthology.orchestra.run.smithi094.stdout: debug osd = 25 2021-02-11T09:42:46.539 INFO:teuthology.orchestra.run.smithi094.stdout: osd fast shutdown = False 2021-02-11T09:42:46.539 INFO:teuthology.orchestra.run.smithi094.stdout: rocksdb delete range threshold = 0 2021-02-11T09:42:46.539 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-02-11T09:42:46.539 INFO:teuthology.orchestra.run.smithi094.stdout:[mgr] 2021-02-11T09:42:46.539 INFO:teuthology.orchestra.run.smithi094.stdout: debug ms = 1 2021-02-11T09:42:46.539 INFO:teuthology.orchestra.run.smithi094.stdout: debug mgr = 20 2021-02-11T09:42:46.539 INFO:teuthology.orchestra.run.smithi094.stdout: debug mon = 20 2021-02-11T09:42:46.539 INFO:teuthology.orchestra.run.smithi094.stdout: debug auth = 20 2021-02-11T09:42:46.539 INFO:teuthology.orchestra.run.smithi094.stdout: mon reweight min pgs per osd = 4 2021-02-11T09:42:46.539 INFO:teuthology.orchestra.run.smithi094.stdout: mon reweight min bytes per osd = 10 2021-02-11T09:42:46.539 INFO:teuthology.orchestra.run.smithi094.stdout: mgr/telemetry/nag = false 2021-02-11T09:42:46.539 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-02-11T09:42:46.539 INFO:teuthology.orchestra.run.smithi094.stdout:[mon] 2021-02-11T09:42:46.540 INFO:teuthology.orchestra.run.smithi094.stdout: debug ms = 1 2021-02-11T09:42:46.540 INFO:teuthology.orchestra.run.smithi094.stdout: debug mon = 20 2021-02-11T09:42:46.540 INFO:teuthology.orchestra.run.smithi094.stdout: debug paxos = 20 2021-02-11T09:42:46.540 INFO:teuthology.orchestra.run.smithi094.stdout: debug auth = 20 2021-02-11T09:42:46.540 INFO:teuthology.orchestra.run.smithi094.stdout: mon data avail warn = 5 2021-02-11T09:42:46.540 INFO:teuthology.orchestra.run.smithi094.stdout: mon mgr mkfs grace = 240 2021-02-11T09:42:46.540 INFO:teuthology.orchestra.run.smithi094.stdout: mon reweight min pgs per osd = 4 2021-02-11T09:42:46.540 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd reporter subtree level = osd 2021-02-11T09:42:46.540 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd prime pg temp = true 2021-02-11T09:42:46.540 INFO:teuthology.orchestra.run.smithi094.stdout: mon reweight min bytes per osd = 10 2021-02-11T09:42:46.540 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd crush smoke test = False 2021-02-11T09:42:46.540 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-02-11T09:42:46.540 INFO:teuthology.orchestra.run.smithi094.stdout:[client] 2021-02-11T09:42:46.540 INFO:teuthology.orchestra.run.smithi094.stdout: rgw cache enabled = true 2021-02-11T09:42:46.540 INFO:teuthology.orchestra.run.smithi094.stdout: rgw enable ops log = true 2021-02-11T09:42:46.541 INFO:teuthology.orchestra.run.smithi094.stdout: rgw enable usage log = true 2021-02-11T09:42:46.541 INFO:teuthology.orchestra.run.smithi094.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-02-11T09:42:46.541 INFO:teuthology.orchestra.run.smithi094.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-02-11T09:42:46.541 INFO:teuthology.orchestra.run.smithi094.stdout: debug rgw = 20 2021-02-11T09:42:46.541 INFO:teuthology.orchestra.run.smithi094.stdout: rgw crypt require ssl = False 2021-02-11T09:42:46.541 INFO:teuthology.orchestra.run.smithi094.stdout: rgw crypt s3 kms backend = testing 2021-02-11T09:42:46.541 INFO:teuthology.orchestra.run.smithi094.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-02-11T09:42:46.541 INFO:teuthology.orchestra.run.smithi094.stdout: rgw curl low speed time = 300 2021-02-11T09:42:46.541 INFO:teuthology.orchestra.run.smithi094.stdout: rgw data log num shards = 4 2021-02-11T09:42:46.541 INFO:teuthology.orchestra.run.smithi094.stdout: rgw md log max shards = 4 2021-02-11T09:42:46.541 INFO:teuthology.orchestra.run.smithi094.stdout: rgw sync log trim interval = 0 2021-02-11T09:42:46.541 INFO:teuthology.orchestra.run.smithi094.stdout: rgw sync obj etag verify = True 2021-02-11T09:42:46.541 INFO:teuthology.orchestra.run.smithi094.stdout: setgroup = ceph 2021-02-11T09:42:46.541 INFO:teuthology.orchestra.run.smithi094.stdout: setuser = ceph 2021-02-11T09:42:46.541 INFO:teuthology.orchestra.run.smithi094.stdout:[mon.a] 2021-02-11T09:42:46.542 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.0] 2021-02-11T09:42:46.542 INFO:teuthology.orchestra.run.smithi094.stdout: osd_max_omap_entries_per_request = 10 2021-02-11T09:42:46.542 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.1] 2021-02-11T09:42:46.542 INFO:teuthology.orchestra.run.smithi094.stdout: osd_max_omap_entries_per_request = 1000 2021-02-11T09:42:46.542 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.2] 2021-02-11T09:42:46.542 INFO:teuthology.orchestra.run.smithi094.stdout: osd_max_omap_entries_per_request = 10000 2021-02-11T09:42:46.561 INFO:teuthology.orchestra.run.smithi098.stdout:[global] 2021-02-11T09:42:46.561 INFO:teuthology.orchestra.run.smithi098.stdout: chdir = "" 2021-02-11T09:42:46.561 INFO:teuthology.orchestra.run.smithi098.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-02-11T09:42:46.562 INFO:teuthology.orchestra.run.smithi098.stdout: auth supported = cephx 2021-02-11T09:42:46.562 INFO:teuthology.orchestra.run.smithi098.stdout: 2021-02-11T09:42:46.562 INFO:teuthology.orchestra.run.smithi098.stdout: filestore xattr use omap = true 2021-02-11T09:42:46.562 INFO:teuthology.orchestra.run.smithi098.stdout: 2021-02-11T09:42:46.562 INFO:teuthology.orchestra.run.smithi098.stdout: mon clock drift allowed = 1.000 2021-02-11T09:42:46.562 INFO:teuthology.orchestra.run.smithi098.stdout: 2021-02-11T09:42:46.562 INFO:teuthology.orchestra.run.smithi098.stdout: osd crush chooseleaf type = 0 2021-02-11T09:42:46.562 INFO:teuthology.orchestra.run.smithi098.stdout: auth debug = true 2021-02-11T09:42:46.562 INFO:teuthology.orchestra.run.smithi098.stdout: 2021-02-11T09:42:46.562 INFO:teuthology.orchestra.run.smithi098.stdout: ms die on old message = true 2021-02-11T09:42:46.562 INFO:teuthology.orchestra.run.smithi098.stdout: ms die on bug = true 2021-02-11T09:42:46.562 INFO:teuthology.orchestra.run.smithi098.stdout: 2021-02-11T09:42:46.562 INFO:teuthology.orchestra.run.smithi098.stdout: mon max pg per osd = 10000 # >= luminous 2021-02-11T09:42:46.563 INFO:teuthology.orchestra.run.smithi098.stdout: mon pg warn max object skew = 0 2021-02-11T09:42:46.563 INFO:teuthology.orchestra.run.smithi098.stdout: 2021-02-11T09:42:46.563 INFO:teuthology.orchestra.run.smithi098.stdout: # disable pg_autoscaler by default for new pools 2021-02-11T09:42:46.563 INFO:teuthology.orchestra.run.smithi098.stdout: osd_pool_default_pg_autoscale_mode = off 2021-02-11T09:42:46.563 INFO:teuthology.orchestra.run.smithi098.stdout: 2021-02-11T09:42:46.563 INFO:teuthology.orchestra.run.smithi098.stdout: osd pool default size = 2 2021-02-11T09:42:46.563 INFO:teuthology.orchestra.run.smithi098.stdout: 2021-02-11T09:42:46.563 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd allow primary affinity = true 2021-02-11T09:42:46.563 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd allow pg remap = true 2021-02-11T09:42:46.563 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on legacy crush tunables = false 2021-02-11T09:42:46.563 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on crush straw calc version zero = false 2021-02-11T09:42:46.563 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on no sortbitwise = false 2021-02-11T09:42:46.563 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on osd down out interval zero = false 2021-02-11T09:42:46.563 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on too few osds = false 2021-02-11T09:42:46.564 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-02-11T09:42:46.564 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_pool_no_redundancy = false 2021-02-11T09:42:46.564 INFO:teuthology.orchestra.run.smithi098.stdout: mon_allow_pool_size_one = true 2021-02-11T09:42:46.564 INFO:teuthology.orchestra.run.smithi098.stdout: 2021-02-11T09:42:46.564 INFO:teuthology.orchestra.run.smithi098.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-02-11T09:42:46.564 INFO:teuthology.orchestra.run.smithi098.stdout: 2021-02-11T09:42:46.564 INFO:teuthology.orchestra.run.smithi098.stdout: osd default data pool replay window = 5 2021-02-11T09:42:46.564 INFO:teuthology.orchestra.run.smithi098.stdout: 2021-02-11T09:42:46.564 INFO:teuthology.orchestra.run.smithi098.stdout: mon allow pool delete = true 2021-02-11T09:42:46.564 INFO:teuthology.orchestra.run.smithi098.stdout: 2021-02-11T09:42:46.564 INFO:teuthology.orchestra.run.smithi098.stdout: mon cluster log file level = debug 2021-02-11T09:42:46.564 INFO:teuthology.orchestra.run.smithi098.stdout: debug asserts on shutdown = true 2021-02-11T09:42:46.564 INFO:teuthology.orchestra.run.smithi098.stdout: mon health detail to clog = false 2021-02-11T09:42:46.565 INFO:teuthology.orchestra.run.smithi098.stdout: mon host = 172.21.15.98 2021-02-11T09:42:46.565 INFO:teuthology.orchestra.run.smithi098.stdout: osd heartbeat grace = 40 2021-02-11T09:42:46.565 INFO:teuthology.orchestra.run.smithi098.stdout: fsid = 0880a54d-a917-41e0-b7d6-67af274484d8 2021-02-11T09:42:46.565 INFO:teuthology.orchestra.run.smithi098.stdout: 2021-02-11T09:42:46.565 INFO:teuthology.orchestra.run.smithi098.stdout:[osd] 2021-02-11T09:42:46.565 INFO:teuthology.orchestra.run.smithi098.stdout: osd journal size = 100 2021-02-11T09:42:46.565 INFO:teuthology.orchestra.run.smithi098.stdout: 2021-02-11T09:42:46.565 INFO:teuthology.orchestra.run.smithi098.stdout: osd scrub load threshold = 5.0 2021-02-11T09:42:46.565 INFO:teuthology.orchestra.run.smithi098.stdout: osd scrub max interval = 600 2021-02-11T09:42:46.565 INFO:teuthology.orchestra.run.smithi098.stdout: 2021-02-11T09:42:46.565 INFO:teuthology.orchestra.run.smithi098.stdout: osd recover clone overlap = true 2021-02-11T09:42:46.565 INFO:teuthology.orchestra.run.smithi098.stdout: osd recovery max chunk = 1048576 2021-02-11T09:42:46.565 INFO:teuthology.orchestra.run.smithi098.stdout: 2021-02-11T09:42:46.565 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug shutdown = true 2021-02-11T09:42:46.566 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug op order = true 2021-02-11T09:42:46.566 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug verify stray on activate = true 2021-02-11T09:42:46.566 INFO:teuthology.orchestra.run.smithi098.stdout: 2021-02-11T09:42:46.566 INFO:teuthology.orchestra.run.smithi098.stdout: osd open classes on start = true 2021-02-11T09:42:46.566 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug pg log writeout = true 2021-02-11T09:42:46.566 INFO:teuthology.orchestra.run.smithi098.stdout: 2021-02-11T09:42:46.566 INFO:teuthology.orchestra.run.smithi098.stdout: osd deep scrub update digest min age = 30 2021-02-11T09:42:46.566 INFO:teuthology.orchestra.run.smithi098.stdout: 2021-02-11T09:42:46.566 INFO:teuthology.orchestra.run.smithi098.stdout: osd map max advance = 10 2021-02-11T09:42:46.566 INFO:teuthology.orchestra.run.smithi098.stdout: 2021-02-11T09:42:46.566 INFO:teuthology.orchestra.run.smithi098.stdout: journal zero on create = true 2021-02-11T09:42:46.566 INFO:teuthology.orchestra.run.smithi098.stdout: 2021-02-11T09:42:46.566 INFO:teuthology.orchestra.run.smithi098.stdout: filestore ondisk finisher threads = 3 2021-02-11T09:42:46.567 INFO:teuthology.orchestra.run.smithi098.stdout: filestore apply finisher threads = 3 2021-02-11T09:42:46.567 INFO:teuthology.orchestra.run.smithi098.stdout: 2021-02-11T09:42:46.567 INFO:teuthology.orchestra.run.smithi098.stdout: bdev debug aio = true 2021-02-11T09:42:46.567 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug misdirected ops = true 2021-02-11T09:42:46.567 INFO:teuthology.orchestra.run.smithi098.stdout: debug filestore = 20 2021-02-11T09:42:46.567 INFO:teuthology.orchestra.run.smithi098.stdout: debug journal = 20 2021-02-11T09:42:46.567 INFO:teuthology.orchestra.run.smithi098.stdout: debug ms = 20 2021-02-11T09:42:46.567 INFO:teuthology.orchestra.run.smithi098.stdout: debug osd = 25 2021-02-11T09:42:46.567 INFO:teuthology.orchestra.run.smithi098.stdout: osd fast shutdown = False 2021-02-11T09:42:46.567 INFO:teuthology.orchestra.run.smithi098.stdout: rocksdb delete range threshold = 0 2021-02-11T09:42:46.567 INFO:teuthology.orchestra.run.smithi098.stdout: 2021-02-11T09:42:46.567 INFO:teuthology.orchestra.run.smithi098.stdout:[mgr] 2021-02-11T09:42:46.567 INFO:teuthology.orchestra.run.smithi098.stdout: debug ms = 1 2021-02-11T09:42:46.568 INFO:teuthology.orchestra.run.smithi098.stdout: debug mgr = 20 2021-02-11T09:42:46.568 INFO:teuthology.orchestra.run.smithi098.stdout: debug mon = 20 2021-02-11T09:42:46.568 INFO:teuthology.orchestra.run.smithi098.stdout: debug auth = 20 2021-02-11T09:42:46.568 INFO:teuthology.orchestra.run.smithi098.stdout: mon reweight min pgs per osd = 4 2021-02-11T09:42:46.568 INFO:teuthology.orchestra.run.smithi098.stdout: mon reweight min bytes per osd = 10 2021-02-11T09:42:46.568 INFO:teuthology.orchestra.run.smithi098.stdout: mgr/telemetry/nag = false 2021-02-11T09:42:46.568 INFO:teuthology.orchestra.run.smithi098.stdout: 2021-02-11T09:42:46.568 INFO:teuthology.orchestra.run.smithi098.stdout:[mon] 2021-02-11T09:42:46.568 INFO:teuthology.orchestra.run.smithi098.stdout: debug ms = 1 2021-02-11T09:42:46.568 INFO:teuthology.orchestra.run.smithi098.stdout: debug mon = 20 2021-02-11T09:42:46.568 INFO:teuthology.orchestra.run.smithi098.stdout: debug paxos = 20 2021-02-11T09:42:46.568 INFO:teuthology.orchestra.run.smithi098.stdout: debug auth = 20 2021-02-11T09:42:46.568 INFO:teuthology.orchestra.run.smithi098.stdout: mon data avail warn = 5 2021-02-11T09:42:46.568 INFO:teuthology.orchestra.run.smithi098.stdout: mon mgr mkfs grace = 240 2021-02-11T09:42:46.568 INFO:teuthology.orchestra.run.smithi098.stdout: mon reweight min pgs per osd = 4 2021-02-11T09:42:46.569 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd reporter subtree level = osd 2021-02-11T09:42:46.569 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd prime pg temp = true 2021-02-11T09:42:46.569 INFO:teuthology.orchestra.run.smithi098.stdout: mon reweight min bytes per osd = 10 2021-02-11T09:42:46.569 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd crush smoke test = False 2021-02-11T09:42:46.569 INFO:teuthology.orchestra.run.smithi098.stdout: 2021-02-11T09:42:46.569 INFO:teuthology.orchestra.run.smithi098.stdout:[client] 2021-02-11T09:42:46.569 INFO:teuthology.orchestra.run.smithi098.stdout: rgw cache enabled = true 2021-02-11T09:42:46.569 INFO:teuthology.orchestra.run.smithi098.stdout: rgw enable ops log = true 2021-02-11T09:42:46.569 INFO:teuthology.orchestra.run.smithi098.stdout: rgw enable usage log = true 2021-02-11T09:42:46.569 INFO:teuthology.orchestra.run.smithi098.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-02-11T09:42:46.569 INFO:teuthology.orchestra.run.smithi098.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-02-11T09:42:46.569 INFO:teuthology.orchestra.run.smithi098.stdout: debug rgw = 20 2021-02-11T09:42:46.569 INFO:teuthology.orchestra.run.smithi098.stdout: rgw crypt require ssl = False 2021-02-11T09:42:46.569 INFO:teuthology.orchestra.run.smithi098.stdout: rgw crypt s3 kms backend = testing 2021-02-11T09:42:46.570 INFO:teuthology.orchestra.run.smithi098.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-02-11T09:42:46.570 INFO:teuthology.orchestra.run.smithi098.stdout: rgw curl low speed time = 300 2021-02-11T09:42:46.570 INFO:teuthology.orchestra.run.smithi098.stdout: rgw data log num shards = 4 2021-02-11T09:42:46.570 INFO:teuthology.orchestra.run.smithi098.stdout: rgw md log max shards = 4 2021-02-11T09:42:46.570 INFO:teuthology.orchestra.run.smithi098.stdout: rgw sync log trim interval = 0 2021-02-11T09:42:46.570 INFO:teuthology.orchestra.run.smithi098.stdout: rgw sync obj etag verify = True 2021-02-11T09:42:46.570 INFO:teuthology.orchestra.run.smithi098.stdout: setgroup = ceph 2021-02-11T09:42:46.570 INFO:teuthology.orchestra.run.smithi098.stdout: setuser = ceph 2021-02-11T09:42:46.570 INFO:teuthology.orchestra.run.smithi098.stdout:[mon.a] 2021-02-11T09:42:46.570 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.0] 2021-02-11T09:42:46.570 INFO:teuthology.orchestra.run.smithi098.stdout: osd_max_omap_entries_per_request = 10 2021-02-11T09:42:46.570 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.1] 2021-02-11T09:42:46.570 INFO:teuthology.orchestra.run.smithi098.stdout: osd_max_omap_entries_per_request = 1000 2021-02-11T09:42:46.571 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.2] 2021-02-11T09:42:46.571 INFO:teuthology.orchestra.run.smithi098.stdout: osd_max_omap_entries_per_request = 10000 2021-02-11T09:42:46.571 INFO:tasks.ceph:Creating admin key on c2.mon.a... 2021-02-11T09:42:46.572 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c2.keyring 2021-02-11T09:42:46.660 INFO:tasks.ceph:Copying monmap to all nodes... 2021-02-11T09:42:46.660 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2021-02-11T09:42:46.660 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/etc/ceph/c2.keyring of=/dev/stdout 2021-02-11T09:42:46.667 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2021-02-11T09:42:46.667 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/home/ubuntu/cephtest/c2.monmap of=/dev/stdout 2021-02-11T09:42:46.716 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi094.front.sepia.ceph.com 2021-02-11T09:42:46.716 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2021-02-11T09:42:46.716 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/etc/ceph/c2.keyring 2021-02-11T09:42:46.716 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 0644 /etc/ceph/c2.keyring 2021-02-11T09:42:46.742 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2021-02-11T09:42:46.742 DEBUG:teuthology.orchestra.run.smithi094:> dd of=/home/ubuntu/cephtest/c2.monmap 2021-02-11T09:42:46.788 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi098.front.sepia.ceph.com 2021-02-11T09:42:46.788 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2021-02-11T09:42:46.788 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/ceph/c2.keyring 2021-02-11T09:42:46.788 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0644 /etc/ceph/c2.keyring 2021-02-11T09:42:46.812 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2021-02-11T09:42:46.812 DEBUG:teuthology.orchestra.run.smithi098:> dd of=/home/ubuntu/cephtest/c2.monmap 2021-02-11T09:42:46.860 INFO:tasks.ceph:Setting up mon nodes... 2021-02-11T09:42:46.861 INFO:tasks.ceph:Setting up mgr nodes... 2021-02-11T09:42:46.861 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/mgr/c2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c2-x/keyring 2021-02-11T09:42:46.943 INFO:teuthology.orchestra.run.smithi098.stdout:creating /var/lib/ceph/mgr/c2-x/keyring 2021-02-11T09:42:46.945 INFO:tasks.ceph:Setting up mds nodes... 2021-02-11T09:42:46.945 INFO:tasks.ceph_client:Setting up client nodes... 2021-02-11T09:42:46.945 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c2.client.0.keyring && sudo chmod 0644 /etc/ceph/c2.client.0.keyring 2021-02-11T09:42:47.022 INFO:teuthology.orchestra.run.smithi098.stdout:creating /etc/ceph/c2.client.0.keyring 2021-02-11T09:42:47.034 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c2.client.1.keyring && sudo chmod 0644 /etc/ceph/c2.client.1.keyring 2021-02-11T09:42:47.107 INFO:teuthology.orchestra.run.smithi098.stdout:creating /etc/ceph/c2.client.1.keyring 2021-02-11T09:42:47.119 INFO:tasks.ceph:Running mkfs on osd nodes... 2021-02-11T09:42:47.119 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi094.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi098.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2021-02-11T09:42:47.119 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/osd/c2-0 2021-02-11T09:42:47.176 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2021-02-11T09:42:47.176 INFO:tasks.ceph:role: c2.osd.0 2021-02-11T09:42:47.177 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi098.front.sepia.ceph.com 2021-02-11T09:42:47.177 DEBUG:teuthology.orchestra.run.smithi098:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2021-02-11T09:42:47.694 INFO:teuthology.orchestra.run.smithi098.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2021-02-11T09:42:47.694 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 attr=2, projid32bit=1 2021-02-11T09:42:47.694 INFO:teuthology.orchestra.run.smithi098.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2021-02-11T09:42:47.694 INFO:teuthology.orchestra.run.smithi098.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-02-11T09:42:47.695 INFO:teuthology.orchestra.run.smithi098.stdout: = sunit=0 swidth=0 blks 2021-02-11T09:42:47.695 INFO:teuthology.orchestra.run.smithi098.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-02-11T09:42:47.695 INFO:teuthology.orchestra.run.smithi098.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-02-11T09:42:47.695 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-02-11T09:42:47.695 INFO:teuthology.orchestra.run.smithi098.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-02-11T09:42:47.696 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi098.front.sepia.ceph.com -o noatime 2021-02-11T09:42:47.696 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c2-0 2021-02-11T09:42:47.792 DEBUG:teuthology.orchestra.run.smithi098:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-0 2021-02-11T09:42:47.803 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: /sbin/restorecon: command not found 2021-02-11T09:42:47.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-11T09:42:47.804 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/osd/c2-1 2021-02-11T09:42:47.859 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2021-02-11T09:42:47.859 INFO:tasks.ceph:role: c2.osd.1 2021-02-11T09:42:47.859 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi098.front.sepia.ceph.com 2021-02-11T09:42:47.859 DEBUG:teuthology.orchestra.run.smithi098:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2021-02-11T09:42:48.366 INFO:teuthology.orchestra.run.smithi098.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2021-02-11T09:42:48.367 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 attr=2, projid32bit=1 2021-02-11T09:42:48.367 INFO:teuthology.orchestra.run.smithi098.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2021-02-11T09:42:48.367 INFO:teuthology.orchestra.run.smithi098.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-02-11T09:42:48.367 INFO:teuthology.orchestra.run.smithi098.stdout: = sunit=0 swidth=0 blks 2021-02-11T09:42:48.367 INFO:teuthology.orchestra.run.smithi098.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-02-11T09:42:48.367 INFO:teuthology.orchestra.run.smithi098.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-02-11T09:42:48.367 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-02-11T09:42:48.367 INFO:teuthology.orchestra.run.smithi098.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-02-11T09:42:48.368 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi098.front.sepia.ceph.com -o noatime 2021-02-11T09:42:48.368 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c2-1 2021-02-11T09:42:48.422 DEBUG:teuthology.orchestra.run.smithi098:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-1 2021-02-11T09:42:48.430 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: /sbin/restorecon: command not found 2021-02-11T09:42:48.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-11T09:42:48.431 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/osd/c2-2 2021-02-11T09:42:48.489 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2021-02-11T09:42:48.490 INFO:tasks.ceph:role: c2.osd.2 2021-02-11T09:42:48.490 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi098.front.sepia.ceph.com 2021-02-11T09:42:48.490 DEBUG:teuthology.orchestra.run.smithi098:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2021-02-11T09:42:48.999 INFO:teuthology.orchestra.run.smithi098.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2021-02-11T09:42:48.999 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 attr=2, projid32bit=1 2021-02-11T09:42:48.999 INFO:teuthology.orchestra.run.smithi098.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2021-02-11T09:42:48.999 INFO:teuthology.orchestra.run.smithi098.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-02-11T09:42:48.999 INFO:teuthology.orchestra.run.smithi098.stdout: = sunit=0 swidth=0 blks 2021-02-11T09:42:48.999 INFO:teuthology.orchestra.run.smithi098.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-02-11T09:42:48.999 INFO:teuthology.orchestra.run.smithi098.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-02-11T09:42:48.999 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-02-11T09:42:48.999 INFO:teuthology.orchestra.run.smithi098.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-02-11T09:42:49.001 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi098.front.sepia.ceph.com -o noatime 2021-02-11T09:42:49.001 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c2-2 2021-02-11T09:42:49.043 DEBUG:teuthology.orchestra.run.smithi098:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-2 2021-02-11T09:42:49.097 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: /sbin/restorecon: command not found 2021-02-11T09:42:49.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-11T09:42:49.097 DEBUG:teuthology.orchestra.run.smithi098:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c2.monmap 2021-02-11T09:42:49.188 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:42:49.189+0000 7ff83d470080 -1 auth: error reading file: /var/lib/ceph/osd/c2-0/keyring: can't open /var/lib/ceph/osd/c2-0/keyring: (2) No such file or directory 2021-02-11T09:42:49.189 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:42:49.189+0000 7ff83d470080 -1 created new key in keyring /var/lib/ceph/osd/c2-0/keyring 2021-02-11T09:42:49.189 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:42:49.189+0000 7ff83d470080 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2021-02-11T09:42:49.189 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:42:49.189+0000 7ff83d470080 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2021-02-11T09:42:49.189 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:42:49.189+0000 7ff83d470080 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2021-02-11T09:42:49.197 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:42:49.197+0000 7ff83d470080 -1 bluestore(/var/lib/ceph/osd/c2-0) _read_fsid unparsable uuid 2021-02-11T09:42:51.367 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-0 2021-02-11T09:42:51.383 DEBUG:teuthology.orchestra.run.smithi098:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c2.monmap 2021-02-11T09:42:51.466 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:42:51.465+0000 7f8fb84c8080 -1 auth: error reading file: /var/lib/ceph/osd/c2-1/keyring: can't open /var/lib/ceph/osd/c2-1/keyring: (2) No such file or directory 2021-02-11T09:42:51.466 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:42:51.465+0000 7f8fb84c8080 -1 created new key in keyring /var/lib/ceph/osd/c2-1/keyring 2021-02-11T09:42:51.466 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:42:51.465+0000 7f8fb84c8080 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2021-02-11T09:42:51.466 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:42:51.465+0000 7f8fb84c8080 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2021-02-11T09:42:51.466 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:42:51.465+0000 7f8fb84c8080 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2021-02-11T09:42:51.467 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:42:51.469+0000 7f8fb84c8080 -1 bluestore(/var/lib/ceph/osd/c2-1) _read_fsid unparsable uuid 2021-02-11T09:42:53.638 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-1 2021-02-11T09:42:53.700 DEBUG:teuthology.orchestra.run.smithi098:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c2.monmap 2021-02-11T09:42:53.781 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:42:53.781+0000 7f0b14416080 -1 auth: error reading file: /var/lib/ceph/osd/c2-2/keyring: can't open /var/lib/ceph/osd/c2-2/keyring: (2) No such file or directory 2021-02-11T09:42:53.781 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:42:53.781+0000 7f0b14416080 -1 created new key in keyring /var/lib/ceph/osd/c2-2/keyring 2021-02-11T09:42:53.781 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:42:53.781+0000 7f0b14416080 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2021-02-11T09:42:53.781 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:42:53.781+0000 7f0b14416080 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2021-02-11T09:42:53.781 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:42:53.781+0000 7f0b14416080 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2021-02-11T09:42:53.782 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:42:53.781+0000 7f0b14416080 -1 bluestore(/var/lib/ceph/osd/c2-2) _read_fsid unparsable uuid 2021-02-11T09:42:55.950 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-2 2021-02-11T09:42:56.006 INFO:tasks.ceph:Reading keys from all nodes... 2021-02-11T09:42:56.006 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2021-02-11T09:42:56.006 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/mgr/c2-x/keyring of=/dev/stdout 2021-02-11T09:42:56.061 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2021-02-11T09:42:56.061 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/c2-0/keyring of=/dev/stdout 2021-02-11T09:42:56.117 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2021-02-11T09:42:56.117 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/c2-1/keyring of=/dev/stdout 2021-02-11T09:42:56.173 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2021-02-11T09:42:56.173 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/c2-2/keyring of=/dev/stdout 2021-02-11T09:42:56.229 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2021-02-11T09:42:56.229 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/etc/ceph/c2.client.0.keyring of=/dev/stdout 2021-02-11T09:42:56.276 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2021-02-11T09:42:56.276 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/etc/ceph/c2.client.1.keyring of=/dev/stdout 2021-02-11T09:42:56.324 INFO:tasks.ceph:Adding keys to all mons... 2021-02-11T09:42:56.325 DEBUG:teuthology.orchestra.run.smithi098:> sudo tee -a /etc/ceph/c2.keyring 2021-02-11T09:42:56.379 INFO:teuthology.orchestra.run.smithi098.stdout:[mgr.x] 2021-02-11T09:42:56.379 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQAW/CRg90FyOBAAgCDK4kjRc2hmIwF1/DdFcw== 2021-02-11T09:42:56.379 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.0] 2021-02-11T09:42:56.379 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQAZ/CRg+5xxCxAAEn8PJ/HrsrET4X2/o8FV1Q== 2021-02-11T09:42:56.379 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.1] 2021-02-11T09:42:56.379 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQAb/CRgt0ABHBAAjsrIWLGr7QLfLuV/W0lNxg== 2021-02-11T09:42:56.380 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.2] 2021-02-11T09:42:56.380 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQAd/CRgySzHLhAAu2SZ/d7HiTZFOBDgDEpsFA== 2021-02-11T09:42:56.380 INFO:teuthology.orchestra.run.smithi098.stdout:[client.0] 2021-02-11T09:42:56.380 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQAX/CRgU7KMARAABbD9zW72SZ/HayLf3uE6lQ== 2021-02-11T09:42:56.380 INFO:teuthology.orchestra.run.smithi098.stdout:[client.1] 2021-02-11T09:42:56.380 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQAX/CRgY32kBhAAdcCVeHUAhA5Gr6By5gq0Kw== 2021-02-11T09:42:56.381 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2021-02-11T09:42:56.463 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-11T09:42:56.548 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-11T09:42:56.635 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-11T09:42:56.720 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-02-11T09:42:56.807 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-02-11T09:42:56.896 INFO:tasks.ceph:Running mkfs on mon nodes... 2021-02-11T09:42:56.896 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/mon/c2-a 2021-02-11T09:42:56.908 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c2 --mkfs -i a --monmap /home/ubuntu/cephtest/c2.monmap --keyring /etc/ceph/c2.keyring 2021-02-11T09:42:57.003 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c2-a 2021-02-11T09:42:57.018 DEBUG:teuthology.orchestra.run.smithi098:> rm -- /home/ubuntu/cephtest/c2.monmap 2021-02-11T09:42:57.068 INFO:tasks.ceph:Starting mon daemons in cluster c2... 2021-02-11T09:42:57.068 INFO:tasks.ceph.c2.mon.a:Restarting daemon 2021-02-11T09:42:57.069 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c2 -i a 2021-02-11T09:42:57.112 INFO:tasks.ceph.c2.mon.a:Started 2021-02-11T09:42:57.113 INFO:tasks.ceph:Starting mgr daemons in cluster c2... 2021-02-11T09:42:57.113 INFO:tasks.ceph.c2.mgr.x:Restarting daemon 2021-02-11T09:42:57.113 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c2 -i x 2021-02-11T09:42:57.115 INFO:tasks.ceph.c2.mgr.x:Started 2021-02-11T09:42:57.115 INFO:tasks.ceph:Setting crush tunables to default 2021-02-11T09:42:57.115 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --cluster c2 osd crush tunables default 2021-02-11T09:42:57.539 INFO:teuthology.orchestra.run.smithi098.stderr:adjusted tunables profile to default 2021-02-11T09:42:57.551 INFO:tasks.ceph:Starting osd daemons in cluster c2... 2021-02-11T09:42:57.551 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2021-02-11T09:42:57.551 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/c2-0/fsid of=/dev/stdout 2021-02-11T09:42:57.560 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2021-02-11T09:42:57.560 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/c2-1/fsid of=/dev/stdout 2021-02-11T09:42:57.619 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2021-02-11T09:42:57.619 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/c2-2/fsid of=/dev/stdout 2021-02-11T09:42:57.677 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --cluster c2 osd new 73ab80cd-978c-4fda-9569-020ac3e0ac0c 0 2021-02-11T09:42:57.931 INFO:teuthology.orchestra.run.smithi098.stdout:0 2021-02-11T09:42:57.944 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --cluster c2 osd new 14a77ff0-2e1c-41c0-815a-8c0a87b02477 1 2021-02-11T09:42:58.162 INFO:teuthology.orchestra.run.smithi098.stdout:1 2021-02-11T09:42:58.174 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --cluster c2 osd new 2f71f5ca-5cbb-420e-ab80-7277fb21d403 2 2021-02-11T09:42:58.380 INFO:teuthology.orchestra.run.smithi098.stdout:2 2021-02-11T09:42:58.392 INFO:tasks.ceph.c2.osd.0:Restarting daemon 2021-02-11T09:42:58.392 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 0 2021-02-11T09:42:58.394 INFO:tasks.ceph.c2.osd.0:Started 2021-02-11T09:42:58.394 INFO:tasks.ceph.c2.osd.1:Restarting daemon 2021-02-11T09:42:58.394 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 1 2021-02-11T09:42:58.396 INFO:tasks.ceph.c2.osd.1:Started 2021-02-11T09:42:58.396 INFO:tasks.ceph.c2.osd.2:Restarting daemon 2021-02-11T09:42:58.397 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 2 2021-02-11T09:42:58.399 INFO:tasks.ceph.c2.osd.2:Started 2021-02-11T09:42:58.399 INFO:tasks.ceph:Waiting for OSDs to come up 2021-02-11T09:42:58.702 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2021-02-11T09:42:58.894 INFO:teuthology.misc.health.smithi098.stdout: 2021-02-11T09:42:58.895 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":5,"fsid":"0880a54d-a917-41e0-b7d6-67af274484d8","created":"2021-02-11T09:42:57.243402+0000","modified":"2021-02-11T09:42:58.380553+0000","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"73ab80cd-978c-4fda-9569-020ac3e0ac0c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a77ff0-2e1c-41c0-815a-8c0a87b02477","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2f71f5ca-5cbb-420e-ab80-7277fb21d403","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-02-11T09:42:58.905 DEBUG:teuthology.misc:0 of 3 OSDs are up 2021-02-11T09:42:59.014 INFO:tasks.ceph.c2.osd.1.smithi098.stderr:2021-02-11T09:42:59.013+0000 7f1660dbe080 -1 Falling back to public interface 2021-02-11T09:42:59.015 INFO:tasks.ceph.c2.osd.0.smithi098.stderr:2021-02-11T09:42:59.013+0000 7f5e0e4c0080 -1 Falling back to public interface 2021-02-11T09:42:59.015 INFO:tasks.ceph.c2.osd.2.smithi098.stderr:2021-02-11T09:42:59.013+0000 7efe0e283080 -1 Falling back to public interface 2021-02-11T09:42:59.018 INFO:tasks.ceph.c2.osd.2.smithi098.stderr:2021-02-11T09:42:59.017+0000 7efe08f53700 -1 PosixStack listen unable to listen on v1:0.0.0.0:6818/0: (98) Address already in use 2021-02-11T09:43:01.746 INFO:tasks.ceph.c2.osd.1.smithi098.stderr:2021-02-11T09:43:01.745+0000 7f1660dbe080 -1 osd.1 0 log_to_monitors {default=true} 2021-02-11T09:43:07.850 INFO:tasks.ceph.c2.osd.2.smithi098.stderr:2021-02-11T09:43:01.925+0000 7efe0e283080 -1 osd.2 0 log_to_monitors {default=true} 2021-02-11T09:43:07.850 INFO:tasks.ceph.c2.osd.2.smithi098.stderr:2021-02-11T09:43:03.829+0000 7efe0574c700 -1 osd.2 0 waiting for initial osdmap 2021-02-11T09:43:07.850 INFO:tasks.ceph.c2.osd.0.smithi098.stderr:2021-02-11T09:43:02.013+0000 7f5e0e4c0080 -1 osd.0 0 log_to_monitors {default=true} 2021-02-11T09:43:07.850 INFO:tasks.ceph.c2.osd.0.smithi098.stderr:2021-02-11T09:43:03.829+0000 7f5e05989700 -1 osd.0 0 waiting for initial osdmap 2021-02-11T09:43:07.850 INFO:tasks.ceph.c2.osd.1.smithi098.stderr:2021-02-11T09:43:02.817+0000 7f1659a8a700 -1 osd.1 0 waiting for initial osdmap 2021-02-11T09:43:08.151 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2021-02-11T09:43:08.451 INFO:teuthology.misc.health.smithi098.stdout: 2021-02-11T09:43:08.451 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":12,"fsid":"0880a54d-a917-41e0-b7d6-67af274484d8","created":"2021-02-11T09:42:57.243402+0000","modified":"2021-02-11T09:43:07.845224+0000","last_up_change":"2021-02-11T09:43:04.825732+0000","last_in_change":"2021-02-11T09:43:04.825732+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-02-11T09:43:04.836347+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"73ab80cd-978c-4fda-9569-020ac3e0ac0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":14447},{"type":"v1","addr":"172.21.15.98:6808","nonce":14447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":14447},{"type":"v1","addr":"172.21.15.98:6813","nonce":14447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6821","nonce":14447},{"type":"v1","addr":"172.21.15.98:6823","nonce":14447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":14447},{"type":"v1","addr":"172.21.15.98:6818","nonce":14447}]},"public_addr":"172.21.15.98:6808/14447","cluster_addr":"172.21.15.98:6813/14447","heartbeat_back_addr":"172.21.15.98:6823/14447","heartbeat_front_addr":"172.21.15.98:6818/14447","state":["exists","up"]},{"osd":1,"uuid":"14a77ff0-2e1c-41c0-815a-8c0a87b02477","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":14448},{"type":"v1","addr":"172.21.15.98:6801","nonce":14448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":14448},{"type":"v1","addr":"172.21.15.98:6803","nonce":14448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":14448},{"type":"v1","addr":"172.21.15.98:6814","nonce":14448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":14448},{"type":"v1","addr":"172.21.15.98:6807","nonce":14448}]},"public_addr":"172.21.15.98:6801/14448","cluster_addr":"172.21.15.98:6803/14448","heartbeat_back_addr":"172.21.15.98:6814/14448","heartbeat_front_addr":"172.21.15.98:6807/14448","state":["exists","up"]},{"osd":2,"uuid":"2f71f5ca-5cbb-420e-ab80-7277fb21d403","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6805","nonce":14449},{"type":"v1","addr":"172.21.15.98:6809","nonce":14449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":14449},{"type":"v1","addr":"172.21.15.98:6815","nonce":14449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":14449},{"type":"v1","addr":"172.21.15.98:6822","nonce":14449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6817","nonce":14449},{"type":"v1","addr":"172.21.15.98:6819","nonce":14449}]},"public_addr":"172.21.15.98:6809/14449","cluster_addr":"172.21.15.98:6815/14449","heartbeat_back_addr":"172.21.15.98:6822/14449","heartbeat_front_addr":"172.21.15.98:6819/14449","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T09:43:02.984492+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T09:43:02.789861+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T09:43:02.946888+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-02-11T09:43:08.464 DEBUG:teuthology.misc:3 of 3 OSDs are up 2021-02-11T09:43:08.464 INFO:tasks.ceph:Creating RBD pool 2021-02-11T09:43:08.464 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --cluster c2 osd pool create rbd 8 2021-02-11T09:43:08.863 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'rbd' created 2021-02-11T09:43:08.875 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --cluster c2 osd pool application enable rbd rbd --yes-i-really-mean-it 2021-02-11T09:43:09.871 INFO:teuthology.orchestra.run.smithi098.stderr:enabled application 'rbd' on pool 'rbd' 2021-02-11T09:43:09.883 INFO:tasks.ceph:Starting mds daemons in cluster c2... 2021-02-11T09:43:09.883 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2021-02-11T09:43:09.884 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2021-02-11T09:43:10.173 INFO:teuthology.orchestra.run.smithi098.stdout: 2021-02-11T09:43:10.173 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":14,"fsid":"0880a54d-a917-41e0-b7d6-67af274484d8","created":"2021-02-11T09:42:57.243402+0000","modified":"2021-02-11T09:43:09.854202+0000","last_up_change":"2021-02-11T09:43:04.825732+0000","last_in_change":"2021-02-11T09:43:04.825732+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-02-11T09:43:04.836347+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-02-11T09:43:08.767274+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"73ab80cd-978c-4fda-9569-020ac3e0ac0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":14447},{"type":"v1","addr":"172.21.15.98:6808","nonce":14447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":14447},{"type":"v1","addr":"172.21.15.98:6813","nonce":14447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6821","nonce":14447},{"type":"v1","addr":"172.21.15.98:6823","nonce":14447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":14447},{"type":"v1","addr":"172.21.15.98:6818","nonce":14447}]},"public_addr":"172.21.15.98:6808/14447","cluster_addr":"172.21.15.98:6813/14447","heartbeat_back_addr":"172.21.15.98:6823/14447","heartbeat_front_addr":"172.21.15.98:6818/14447","state":["exists","up"]},{"osd":1,"uuid":"14a77ff0-2e1c-41c0-815a-8c0a87b02477","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":14448},{"type":"v1","addr":"172.21.15.98:6801","nonce":14448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":14448},{"type":"v1","addr":"172.21.15.98:6803","nonce":14448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":14448},{"type":"v1","addr":"172.21.15.98:6814","nonce":14448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":14448},{"type":"v1","addr":"172.21.15.98:6807","nonce":14448}]},"public_addr":"172.21.15.98:6801/14448","cluster_addr":"172.21.15.98:6803/14448","heartbeat_back_addr":"172.21.15.98:6814/14448","heartbeat_front_addr":"172.21.15.98:6807/14448","state":["exists","up"]},{"osd":2,"uuid":"2f71f5ca-5cbb-420e-ab80-7277fb21d403","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6805","nonce":14449},{"type":"v1","addr":"172.21.15.98:6809","nonce":14449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":14449},{"type":"v1","addr":"172.21.15.98:6815","nonce":14449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":14449},{"type":"v1","addr":"172.21.15.98:6822","nonce":14449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6817","nonce":14449},{"type":"v1","addr":"172.21.15.98:6819","nonce":14449}]},"public_addr":"172.21.15.98:6809/14449","cluster_addr":"172.21.15.98:6815/14449","heartbeat_back_addr":"172.21.15.98:6822/14449","heartbeat_front_addr":"172.21.15.98:6819/14449","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T09:43:02.984492+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T09:43:02.789861+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T09:43:02.946888+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-02-11T09:43:10.195 INFO:tasks.ceph.ceph_manager.c2:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2021-02-11T09:43:04.836347+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '12', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2021-02-11T09:43:08.767274+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '14', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-11T09:43:10.195 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd pool get device_health_metrics pg_num 2021-02-11T09:43:10.495 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2021-02-11T09:43:10.508 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd pool get rbd pg_num 2021-02-11T09:43:10.805 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 8 2021-02-11T09:43:10.819 INFO:tasks.ceph:Waiting until c2 daemons up and pgs clean... 2021-02-11T09:43:10.819 INFO:tasks.ceph.ceph_manager.c2:waiting for mgr available 2021-02-11T09:43:10.820 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 mgr dump --format=json 2021-02-11T09:43:11.152 INFO:teuthology.orchestra.run.smithi098.stdout: 2021-02-11T09:43:11.165 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":4,"active_gid":4099,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":14264},{"type":"v1","addr":"172.21.15.98:6825","nonce":14264}]},"active_addr":"172.21.15.98:6825/14264","active_change":"2021-02-11T09:43:02.124841+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading remoto library:No module named 'remoto'","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"haproxy","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.98:0","nonce":849074321}]},{"addrvec":[{"type":"v2","addr":"172.21.15.98:0","nonce":1313758290}]}]}} 2021-02-11T09:43:11.167 INFO:tasks.ceph.ceph_manager.c2:mgr available! 2021-02-11T09:43:11.167 INFO:tasks.ceph.ceph_manager.c2:waiting for all up 2021-02-11T09:43:11.167 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2021-02-11T09:43:11.462 INFO:teuthology.orchestra.run.smithi098.stdout: 2021-02-11T09:43:11.462 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":15,"fsid":"0880a54d-a917-41e0-b7d6-67af274484d8","created":"2021-02-11T09:42:57.243402+0000","modified":"2021-02-11T09:43:10.866102+0000","last_up_change":"2021-02-11T09:43:04.825732+0000","last_in_change":"2021-02-11T09:43:04.825732+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-02-11T09:43:04.836347+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-02-11T09:43:08.767274+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"73ab80cd-978c-4fda-9569-020ac3e0ac0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":14447},{"type":"v1","addr":"172.21.15.98:6808","nonce":14447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":14447},{"type":"v1","addr":"172.21.15.98:6813","nonce":14447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6821","nonce":14447},{"type":"v1","addr":"172.21.15.98:6823","nonce":14447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":14447},{"type":"v1","addr":"172.21.15.98:6818","nonce":14447}]},"public_addr":"172.21.15.98:6808/14447","cluster_addr":"172.21.15.98:6813/14447","heartbeat_back_addr":"172.21.15.98:6823/14447","heartbeat_front_addr":"172.21.15.98:6818/14447","state":["exists","up"]},{"osd":1,"uuid":"14a77ff0-2e1c-41c0-815a-8c0a87b02477","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":14448},{"type":"v1","addr":"172.21.15.98:6801","nonce":14448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":14448},{"type":"v1","addr":"172.21.15.98:6803","nonce":14448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":14448},{"type":"v1","addr":"172.21.15.98:6814","nonce":14448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":14448},{"type":"v1","addr":"172.21.15.98:6807","nonce":14448}]},"public_addr":"172.21.15.98:6801/14448","cluster_addr":"172.21.15.98:6803/14448","heartbeat_back_addr":"172.21.15.98:6814/14448","heartbeat_front_addr":"172.21.15.98:6807/14448","state":["exists","up"]},{"osd":2,"uuid":"2f71f5ca-5cbb-420e-ab80-7277fb21d403","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6805","nonce":14449},{"type":"v1","addr":"172.21.15.98:6809","nonce":14449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":14449},{"type":"v1","addr":"172.21.15.98:6815","nonce":14449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":14449},{"type":"v1","addr":"172.21.15.98:6822","nonce":14449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6817","nonce":14449},{"type":"v1","addr":"172.21.15.98:6819","nonce":14449}]},"public_addr":"172.21.15.98:6809/14449","cluster_addr":"172.21.15.98:6815/14449","heartbeat_back_addr":"172.21.15.98:6822/14449","heartbeat_front_addr":"172.21.15.98:6819/14449","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T09:43:02.984492+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T09:43:02.789861+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T09:43:02.946888+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-02-11T09:43:11.476 INFO:tasks.ceph.ceph_manager.c2:all up! 2021-02-11T09:43:11.477 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2021-02-11T09:43:11.764 INFO:teuthology.orchestra.run.smithi098.stdout: 2021-02-11T09:43:11.764 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":15,"fsid":"0880a54d-a917-41e0-b7d6-67af274484d8","created":"2021-02-11T09:42:57.243402+0000","modified":"2021-02-11T09:43:10.866102+0000","last_up_change":"2021-02-11T09:43:04.825732+0000","last_in_change":"2021-02-11T09:43:04.825732+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-02-11T09:43:04.836347+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-02-11T09:43:08.767274+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"73ab80cd-978c-4fda-9569-020ac3e0ac0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":14447},{"type":"v1","addr":"172.21.15.98:6808","nonce":14447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":14447},{"type":"v1","addr":"172.21.15.98:6813","nonce":14447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6821","nonce":14447},{"type":"v1","addr":"172.21.15.98:6823","nonce":14447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":14447},{"type":"v1","addr":"172.21.15.98:6818","nonce":14447}]},"public_addr":"172.21.15.98:6808/14447","cluster_addr":"172.21.15.98:6813/14447","heartbeat_back_addr":"172.21.15.98:6823/14447","heartbeat_front_addr":"172.21.15.98:6818/14447","state":["exists","up"]},{"osd":1,"uuid":"14a77ff0-2e1c-41c0-815a-8c0a87b02477","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":14448},{"type":"v1","addr":"172.21.15.98:6801","nonce":14448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":14448},{"type":"v1","addr":"172.21.15.98:6803","nonce":14448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":14448},{"type":"v1","addr":"172.21.15.98:6814","nonce":14448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":14448},{"type":"v1","addr":"172.21.15.98:6807","nonce":14448}]},"public_addr":"172.21.15.98:6801/14448","cluster_addr":"172.21.15.98:6803/14448","heartbeat_back_addr":"172.21.15.98:6814/14448","heartbeat_front_addr":"172.21.15.98:6807/14448","state":["exists","up"]},{"osd":2,"uuid":"2f71f5ca-5cbb-420e-ab80-7277fb21d403","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6805","nonce":14449},{"type":"v1","addr":"172.21.15.98:6809","nonce":14449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":14449},{"type":"v1","addr":"172.21.15.98:6815","nonce":14449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":14449},{"type":"v1","addr":"172.21.15.98:6822","nonce":14449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6817","nonce":14449},{"type":"v1","addr":"172.21.15.98:6819","nonce":14449}]},"public_addr":"172.21.15.98:6809/14449","cluster_addr":"172.21.15.98:6815/14449","heartbeat_back_addr":"172.21.15.98:6822/14449","heartbeat_front_addr":"172.21.15.98:6819/14449","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T09:43:02.984492+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T09:43:02.789861+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T09:43:02.946888+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-02-11T09:43:11.778 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.0 flush_pg_stats 2021-02-11T09:43:11.947 INFO:teuthology.orchestra.run.smithi098.stdout:38654705668 2021-02-11T09:43:11.947 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.1 flush_pg_stats 2021-02-11T09:43:12.116 INFO:teuthology.orchestra.run.smithi098.stdout:34359738372 2021-02-11T09:43:12.117 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.2 flush_pg_stats 2021-02-11T09:43:12.278 INFO:teuthology.orchestra.run.smithi098.stdout:38654705668 2021-02-11T09:43:12.278 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2021-02-11T09:43:12.576 INFO:teuthology.orchestra.run.smithi098.stdout:38654705666 2021-02-11T09:43:12.590 INFO:tasks.ceph.ceph_manager.c2:need seq 38654705668 got 38654705666 for osd.0 2021-02-11T09:43:13.591 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2021-02-11T09:43:13.903 INFO:teuthology.orchestra.run.smithi098.stdout:38654705668 2021-02-11T09:43:13.921 INFO:tasks.ceph.ceph_manager.c2:need seq 38654705668 got 38654705668 for osd.0 2021-02-11T09:43:13.922 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2021-02-11T09:43:14.238 INFO:teuthology.orchestra.run.smithi098.stdout:34359738372 2021-02-11T09:43:14.251 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738372 got 34359738372 for osd.1 2021-02-11T09:43:14.252 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2021-02-11T09:43:14.546 INFO:teuthology.orchestra.run.smithi098.stdout:38654705668 2021-02-11T09:43:14.565 INFO:tasks.ceph.ceph_manager.c2:need seq 38654705668 got 38654705668 for osd.2 2021-02-11T09:43:14.565 INFO:tasks.ceph.ceph_manager.c2:waiting for clean 2021-02-11T09:43:14.565 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2021-02-11T09:43:14.854 INFO:teuthology.orchestra.run.smithi098.stdout: 2021-02-11T09:43:14.856 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2021-02-11T09:43:14.868 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2021-02-11T09:43:14.134235+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":14792,"kb_used_data":368,"kb_used_omap":0,"kb_used_meta":14400,"kb_avail":314558008,"statfs":{"total":322122547200,"available":322107400192,"internally_reserved":0,"allocated":376832,"data_stored":105519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":14745600},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"8","reported_epoch":"14","state":"active+clean","last_fresh":"2021-02-11T09:43:09.872301+0000","last_change":"2021-02-11T09:43:09.872301+0000","last_active":"2021-02-11T09:43:09.872301+0000","last_peered":"2021-02-11T09:43:09.872301+0000","last_clean":"2021-02-11T09:43:09.872301+0000","last_became_active":"2021-02-11T09:43:09.871995+0000","last_became_peered":"2021-02-11T09:43:09.871995+0000","last_unstale":"2021-02-11T09:43:09.872301+0000","last_undegraded":"2021-02-11T09:43:09.872301+0000","last_fullsized":"2021-02-11T09:43:09.872301+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T09:43:08.847309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T09:43:08.847309+0000","last_clean_scrub_stamp":"2021-02-11T09:43:08.847309+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"8","reported_epoch":"14","state":"active+clean","last_fresh":"2021-02-11T09:43:09.872126+0000","last_change":"2021-02-11T09:43:09.872126+0000","last_active":"2021-02-11T09:43:09.872126+0000","last_peered":"2021-02-11T09:43:09.872126+0000","last_clean":"2021-02-11T09:43:09.872126+0000","last_became_active":"2021-02-11T09:43:09.871766+0000","last_became_peered":"2021-02-11T09:43:09.871766+0000","last_unstale":"2021-02-11T09:43:09.872126+0000","last_undegraded":"2021-02-11T09:43:09.872126+0000","last_fullsized":"2021-02-11T09:43:09.872126+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T09:43:08.847309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T09:43:08.847309+0000","last_clean_scrub_stamp":"2021-02-11T09:43:08.847309+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"8","reported_epoch":"14","state":"active+clean","last_fresh":"2021-02-11T09:43:09.870239+0000","last_change":"2021-02-11T09:43:09.870239+0000","last_active":"2021-02-11T09:43:09.870239+0000","last_peered":"2021-02-11T09:43:09.870239+0000","last_clean":"2021-02-11T09:43:09.870239+0000","last_became_active":"2021-02-11T09:43:09.869271+0000","last_became_peered":"2021-02-11T09:43:09.869271+0000","last_unstale":"2021-02-11T09:43:09.870239+0000","last_undegraded":"2021-02-11T09:43:09.870239+0000","last_fullsized":"2021-02-11T09:43:09.870239+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T09:43:08.847309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T09:43:08.847309+0000","last_clean_scrub_stamp":"2021-02-11T09:43:08.847309+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"8","reported_epoch":"14","state":"active+clean","last_fresh":"2021-02-11T09:43:09.873751+0000","last_change":"2021-02-11T09:43:09.873751+0000","last_active":"2021-02-11T09:43:09.873751+0000","last_peered":"2021-02-11T09:43:09.873751+0000","last_clean":"2021-02-11T09:43:09.873751+0000","last_became_active":"2021-02-11T09:43:09.873443+0000","last_became_peered":"2021-02-11T09:43:09.873443+0000","last_unstale":"2021-02-11T09:43:09.873751+0000","last_undegraded":"2021-02-11T09:43:09.873751+0000","last_fullsized":"2021-02-11T09:43:09.873751+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T09:43:08.847309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T09:43:08.847309+0000","last_clean_scrub_stamp":"2021-02-11T09:43:08.847309+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"10","reported_epoch":"13","state":"active+clean","last_fresh":"2021-02-11T09:43:08.858811+0000","last_change":"2021-02-11T09:43:07.057980+0000","last_active":"2021-02-11T09:43:08.858811+0000","last_peered":"2021-02-11T09:43:08.858811+0000","last_clean":"2021-02-11T09:43:08.858811+0000","last_became_active":"2021-02-11T09:43:07.057453+0000","last_became_peered":"2021-02-11T09:43:07.057453+0000","last_unstale":"2021-02-11T09:43:08.858811+0000","last_undegraded":"2021-02-11T09:43:08.858811+0000","last_fullsized":"2021-02-11T09:43:08.858811+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T09:43:05.827401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T09:43:05.827401+0000","last_clean_scrub_stamp":"2021-02-11T09:43:05.827401+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"8","reported_epoch":"14","state":"active+clean","last_fresh":"2021-02-11T09:43:09.869718+0000","last_change":"2021-02-11T09:43:09.869718+0000","last_active":"2021-02-11T09:43:09.869718+0000","last_peered":"2021-02-11T09:43:09.869718+0000","last_clean":"2021-02-11T09:43:09.869718+0000","last_became_active":"2021-02-11T09:43:09.869072+0000","last_became_peered":"2021-02-11T09:43:09.869072+0000","last_unstale":"2021-02-11T09:43:09.869718+0000","last_undegraded":"2021-02-11T09:43:09.869718+0000","last_fullsized":"2021-02-11T09:43:09.869718+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T09:43:08.847309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T09:43:08.847309+0000","last_clean_scrub_stamp":"2021-02-11T09:43:08.847309+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"8","reported_epoch":"14","state":"active+clean","last_fresh":"2021-02-11T09:43:09.873360+0000","last_change":"2021-02-11T09:43:09.873360+0000","last_active":"2021-02-11T09:43:09.873360+0000","last_peered":"2021-02-11T09:43:09.873360+0000","last_clean":"2021-02-11T09:43:09.873360+0000","last_became_active":"2021-02-11T09:43:09.873044+0000","last_became_peered":"2021-02-11T09:43:09.873044+0000","last_unstale":"2021-02-11T09:43:09.873360+0000","last_undegraded":"2021-02-11T09:43:09.873360+0000","last_fullsized":"2021-02-11T09:43:09.873360+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T09:43:08.847309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T09:43:08.847309+0000","last_clean_scrub_stamp":"2021-02-11T09:43:08.847309+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2021-02-11T09:43:09.873527+0000","last_change":"2021-02-11T09:43:09.872129+0000","last_active":"2021-02-11T09:43:09.873527+0000","last_peered":"2021-02-11T09:43:09.873527+0000","last_clean":"2021-02-11T09:43:09.873527+0000","last_became_active":"2021-02-11T09:43:09.871846+0000","last_became_peered":"2021-02-11T09:43:09.871846+0000","last_unstale":"2021-02-11T09:43:09.873527+0000","last_undegraded":"2021-02-11T09:43:09.873527+0000","last_fullsized":"2021-02-11T09:43:09.873527+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T09:43:08.847309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T09:43:08.847309+0000","last_clean_scrub_stamp":"2021-02-11T09:43:08.847309+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"8","reported_epoch":"14","state":"active+clean","last_fresh":"2021-02-11T09:43:09.869888+0000","last_change":"2021-02-11T09:43:09.869888+0000","last_active":"2021-02-11T09:43:09.869888+0000","last_peered":"2021-02-11T09:43:09.869888+0000","last_clean":"2021-02-11T09:43:09.869888+0000","last_became_active":"2021-02-11T09:43:09.869558+0000","last_became_peered":"2021-02-11T09:43:09.869558+0000","last_unstale":"2021-02-11T09:43:09.869888+0000","last_undegraded":"2021-02-11T09:43:09.869888+0000","last_fullsized":"2021-02-11T09:43:09.869888+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T09:43:08.847309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T09:43:08.847309+0000","last_clean_scrub_stamp":"2021-02-11T09:43:08.847309+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705668,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4928,"kb_used_data":120,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852672,"statfs":{"total":107374182400,"available":107369136128,"internally_reserved":0,"allocated":122880,"data_stored":34002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96599999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.323}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4928,"kb_used_data":120,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852672,"statfs":{"total":107374182400,"available":107369136128,"internally_reserved":0,"allocated":122880,"data_stored":34002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.845}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.496}]}]},{"osd":0,"up_from":9,"seq":38654705668,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4936,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852664,"statfs":{"total":107374182400,"available":107369127936,"internally_reserved":0,"allocated":131072,"data_stored":37515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.133}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88700000000000001}]}]}]}} 2021-02-11T09:43:14.869 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2021-02-11T09:43:15.159 INFO:teuthology.orchestra.run.smithi098.stdout: 2021-02-11T09:43:15.162 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2021-02-11T09:43:15.173 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2021-02-11T09:43:14.134235+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":14792,"kb_used_data":368,"kb_used_omap":0,"kb_used_meta":14400,"kb_avail":314558008,"statfs":{"total":322122547200,"available":322107400192,"internally_reserved":0,"allocated":376832,"data_stored":105519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":14745600},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"8","reported_epoch":"14","state":"active+clean","last_fresh":"2021-02-11T09:43:09.872301+0000","last_change":"2021-02-11T09:43:09.872301+0000","last_active":"2021-02-11T09:43:09.872301+0000","last_peered":"2021-02-11T09:43:09.872301+0000","last_clean":"2021-02-11T09:43:09.872301+0000","last_became_active":"2021-02-11T09:43:09.871995+0000","last_became_peered":"2021-02-11T09:43:09.871995+0000","last_unstale":"2021-02-11T09:43:09.872301+0000","last_undegraded":"2021-02-11T09:43:09.872301+0000","last_fullsized":"2021-02-11T09:43:09.872301+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T09:43:08.847309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T09:43:08.847309+0000","last_clean_scrub_stamp":"2021-02-11T09:43:08.847309+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"8","reported_epoch":"14","state":"active+clean","last_fresh":"2021-02-11T09:43:09.872126+0000","last_change":"2021-02-11T09:43:09.872126+0000","last_active":"2021-02-11T09:43:09.872126+0000","last_peered":"2021-02-11T09:43:09.872126+0000","last_clean":"2021-02-11T09:43:09.872126+0000","last_became_active":"2021-02-11T09:43:09.871766+0000","last_became_peered":"2021-02-11T09:43:09.871766+0000","last_unstale":"2021-02-11T09:43:09.872126+0000","last_undegraded":"2021-02-11T09:43:09.872126+0000","last_fullsized":"2021-02-11T09:43:09.872126+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T09:43:08.847309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T09:43:08.847309+0000","last_clean_scrub_stamp":"2021-02-11T09:43:08.847309+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"8","reported_epoch":"14","state":"active+clean","last_fresh":"2021-02-11T09:43:09.870239+0000","last_change":"2021-02-11T09:43:09.870239+0000","last_active":"2021-02-11T09:43:09.870239+0000","last_peered":"2021-02-11T09:43:09.870239+0000","last_clean":"2021-02-11T09:43:09.870239+0000","last_became_active":"2021-02-11T09:43:09.869271+0000","last_became_peered":"2021-02-11T09:43:09.869271+0000","last_unstale":"2021-02-11T09:43:09.870239+0000","last_undegraded":"2021-02-11T09:43:09.870239+0000","last_fullsized":"2021-02-11T09:43:09.870239+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T09:43:08.847309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T09:43:08.847309+0000","last_clean_scrub_stamp":"2021-02-11T09:43:08.847309+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"8","reported_epoch":"14","state":"active+clean","last_fresh":"2021-02-11T09:43:09.873751+0000","last_change":"2021-02-11T09:43:09.873751+0000","last_active":"2021-02-11T09:43:09.873751+0000","last_peered":"2021-02-11T09:43:09.873751+0000","last_clean":"2021-02-11T09:43:09.873751+0000","last_became_active":"2021-02-11T09:43:09.873443+0000","last_became_peered":"2021-02-11T09:43:09.873443+0000","last_unstale":"2021-02-11T09:43:09.873751+0000","last_undegraded":"2021-02-11T09:43:09.873751+0000","last_fullsized":"2021-02-11T09:43:09.873751+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T09:43:08.847309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T09:43:08.847309+0000","last_clean_scrub_stamp":"2021-02-11T09:43:08.847309+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"10","reported_epoch":"13","state":"active+clean","last_fresh":"2021-02-11T09:43:08.858811+0000","last_change":"2021-02-11T09:43:07.057980+0000","last_active":"2021-02-11T09:43:08.858811+0000","last_peered":"2021-02-11T09:43:08.858811+0000","last_clean":"2021-02-11T09:43:08.858811+0000","last_became_active":"2021-02-11T09:43:07.057453+0000","last_became_peered":"2021-02-11T09:43:07.057453+0000","last_unstale":"2021-02-11T09:43:08.858811+0000","last_undegraded":"2021-02-11T09:43:08.858811+0000","last_fullsized":"2021-02-11T09:43:08.858811+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T09:43:05.827401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T09:43:05.827401+0000","last_clean_scrub_stamp":"2021-02-11T09:43:05.827401+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"8","reported_epoch":"14","state":"active+clean","last_fresh":"2021-02-11T09:43:09.869718+0000","last_change":"2021-02-11T09:43:09.869718+0000","last_active":"2021-02-11T09:43:09.869718+0000","last_peered":"2021-02-11T09:43:09.869718+0000","last_clean":"2021-02-11T09:43:09.869718+0000","last_became_active":"2021-02-11T09:43:09.869072+0000","last_became_peered":"2021-02-11T09:43:09.869072+0000","last_unstale":"2021-02-11T09:43:09.869718+0000","last_undegraded":"2021-02-11T09:43:09.869718+0000","last_fullsized":"2021-02-11T09:43:09.869718+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T09:43:08.847309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T09:43:08.847309+0000","last_clean_scrub_stamp":"2021-02-11T09:43:08.847309+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"8","reported_epoch":"14","state":"active+clean","last_fresh":"2021-02-11T09:43:09.873360+0000","last_change":"2021-02-11T09:43:09.873360+0000","last_active":"2021-02-11T09:43:09.873360+0000","last_peered":"2021-02-11T09:43:09.873360+0000","last_clean":"2021-02-11T09:43:09.873360+0000","last_became_active":"2021-02-11T09:43:09.873044+0000","last_became_peered":"2021-02-11T09:43:09.873044+0000","last_unstale":"2021-02-11T09:43:09.873360+0000","last_undegraded":"2021-02-11T09:43:09.873360+0000","last_fullsized":"2021-02-11T09:43:09.873360+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T09:43:08.847309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T09:43:08.847309+0000","last_clean_scrub_stamp":"2021-02-11T09:43:08.847309+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"9","reported_epoch":"14","state":"active+clean","last_fresh":"2021-02-11T09:43:09.873527+0000","last_change":"2021-02-11T09:43:09.872129+0000","last_active":"2021-02-11T09:43:09.873527+0000","last_peered":"2021-02-11T09:43:09.873527+0000","last_clean":"2021-02-11T09:43:09.873527+0000","last_became_active":"2021-02-11T09:43:09.871846+0000","last_became_peered":"2021-02-11T09:43:09.871846+0000","last_unstale":"2021-02-11T09:43:09.873527+0000","last_undegraded":"2021-02-11T09:43:09.873527+0000","last_fullsized":"2021-02-11T09:43:09.873527+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T09:43:08.847309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T09:43:08.847309+0000","last_clean_scrub_stamp":"2021-02-11T09:43:08.847309+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"8","reported_epoch":"14","state":"active+clean","last_fresh":"2021-02-11T09:43:09.869888+0000","last_change":"2021-02-11T09:43:09.869888+0000","last_active":"2021-02-11T09:43:09.869888+0000","last_peered":"2021-02-11T09:43:09.869888+0000","last_clean":"2021-02-11T09:43:09.869888+0000","last_became_active":"2021-02-11T09:43:09.869558+0000","last_became_peered":"2021-02-11T09:43:09.869558+0000","last_unstale":"2021-02-11T09:43:09.869888+0000","last_undegraded":"2021-02-11T09:43:09.869888+0000","last_fullsized":"2021-02-11T09:43:09.869888+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T09:43:08.847309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T09:43:08.847309+0000","last_clean_scrub_stamp":"2021-02-11T09:43:08.847309+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705668,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4928,"kb_used_data":120,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852672,"statfs":{"total":107374182400,"available":107369136128,"internally_reserved":0,"allocated":122880,"data_stored":34002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96599999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.323}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4928,"kb_used_data":120,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852672,"statfs":{"total":107374182400,"available":107369136128,"internally_reserved":0,"allocated":122880,"data_stored":34002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.845}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.496}]}]},{"osd":0,"up_from":9,"seq":38654705668,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4936,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852664,"statfs":{"total":107374182400,"available":107369127936,"internally_reserved":0,"allocated":131072,"data_stored":37515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.133}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88700000000000001}]}]}]}} 2021-02-11T09:43:15.174 INFO:tasks.ceph.ceph_manager.c2:clean! 2021-02-11T09:43:15.174 INFO:tasks.ceph:Waiting until ceph cluster c2 is healthy... 2021-02-11T09:43:15.174 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy 2021-02-11T09:43:15.174 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 health --format=json 2021-02-11T09:43:15.527 INFO:teuthology.orchestra.run.smithi098.stdout: 2021-02-11T09:43:15.540 INFO:teuthology.orchestra.run.smithi098.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2021-02-11T09:43:15.541 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy done 2021-02-11T09:43:15.541 INFO:teuthology.run_tasks:Running task rgw... 2021-02-11T09:43:15.556 DEBUG:tasks.rgw:config is {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}} 2021-02-11T09:43:15.557 DEBUG:tasks.rgw:client list is dict_keys(['c1.client.0', 'c1.client.1', 'c2.client.0', 'c2.client.1']) 2021-02-11T09:43:15.557 INFO:tasks.rgw:Creating data pools 2021-02-11T09:43:15.557 DEBUG:tasks.rgw:Obtaining remote for client c1.client.0 2021-02-11T09:43:15.557 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2021-02-11T09:43:16.257 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'default.rgw.buckets.data' created 2021-02-11T09:43:16.272 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2021-02-11T09:43:17.274 INFO:teuthology.orchestra.run.smithi094.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2021-02-11T09:43:17.293 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2021-02-11T09:43:18.269 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'default.rgw.buckets.index' created 2021-02-11T09:43:18.284 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2021-02-11T09:43:19.277 INFO:teuthology.orchestra.run.smithi094.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2021-02-11T09:43:19.300 DEBUG:tasks.rgw:Obtaining remote for client c1.client.1 2021-02-11T09:43:19.300 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2021-02-11T09:43:19.589 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'default.rgw.buckets.data' already exists 2021-02-11T09:43:19.599 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2021-02-11T09:43:20.270 INFO:teuthology.orchestra.run.smithi094.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2021-02-11T09:43:20.283 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2021-02-11T09:43:20.585 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'default.rgw.buckets.index' already exists 2021-02-11T09:43:20.596 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2021-02-11T09:43:21.275 INFO:teuthology.orchestra.run.smithi094.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2021-02-11T09:43:21.289 DEBUG:tasks.rgw:Obtaining remote for client c2.client.0 2021-02-11T09:43:21.290 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2021-02-11T09:43:22.254 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'default.rgw.buckets.data' created 2021-02-11T09:43:22.275 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2021-02-11T09:43:23.272 INFO:teuthology.orchestra.run.smithi098.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2021-02-11T09:43:23.297 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2021-02-11T09:43:24.273 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'default.rgw.buckets.index' created 2021-02-11T09:43:24.290 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2021-02-11T09:43:25.295 INFO:teuthology.orchestra.run.smithi098.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2021-02-11T09:43:25.316 DEBUG:tasks.rgw:Obtaining remote for client c2.client.1 2021-02-11T09:43:25.316 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2021-02-11T09:43:25.605 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'default.rgw.buckets.data' already exists 2021-02-11T09:43:25.615 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2021-02-11T09:43:26.277 INFO:teuthology.orchestra.run.smithi098.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2021-02-11T09:43:26.291 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2021-02-11T09:43:26.593 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'default.rgw.buckets.index' already exists 2021-02-11T09:43:26.604 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2021-02-11T09:43:27.283 INFO:teuthology.orchestra.run.smithi098.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2021-02-11T09:43:27.296 DEBUG:tasks.rgw:Pools created 2021-02-11T09:43:27.296 INFO:tasks.rgw:Configuring compression type = random 2021-02-11T09:43:27.296 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'list'] 2021-02-11T09:43:27.296 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'list'] 2021-02-11T09:43:27.297 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user list 2021-02-11T09:43:27.332 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:43:27.332 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:43:27.342 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:27.339+0000 7f1293ac5080 20 rados->read ofs=0 len=0 2021-02-11T09:43:29.230 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:29.226+0000 7f1293ac5080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:43:29.230 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:29.226+0000 7f1293ac5080 20 rados->read ofs=0 len=0 2021-02-11T09:43:29.234 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:29.230+0000 7f1293ac5080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:43:29.234 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:29.230+0000 7f1293ac5080 20 rados->read ofs=0 len=0 2021-02-11T09:43:29.234 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:29.230+0000 7f1293ac5080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:43:29.234 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:29.230+0000 7f1293ac5080 20 rados->read ofs=0 len=0 2021-02-11T09:43:29.235 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:29.234+0000 7f1293ac5080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:43:29.256 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:29.254+0000 7f1293ac5080 20 rados->read ofs=0 len=0 2021-02-11T09:43:29.256 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:29.254+0000 7f1293ac5080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:43:29.256 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:29.254+0000 7f1293ac5080 20 rados->read ofs=0 len=0 2021-02-11T09:43:29.256 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:29.254+0000 7f1293ac5080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:43:29.257 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:29.254+0000 7f1293ac5080 20 rados->read ofs=0 len=0 2021-02-11T09:43:29.257 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:29.254+0000 7f1293ac5080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:43:29.257 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:29.254+0000 7f1293ac5080 10 cannot find current period zonegroup using local zonegroup 2021-02-11T09:43:29.257 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:29.254+0000 7f1293ac5080 20 rados->read ofs=0 len=0 2021-02-11T09:43:29.258 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:29.254+0000 7f1293ac5080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:43:29.258 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:29.254+0000 7f1293ac5080 20 rados->read ofs=0 len=0 2021-02-11T09:43:29.258 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:29.254+0000 7f1293ac5080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:43:29.258 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:29.254+0000 7f1293ac5080 10 Creating default zonegroup 2021-02-11T09:43:29.258 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:29.254+0000 7f1293ac5080 20 get_system_obj_state: rctx=0x7ffdc7256640 obj=:.pools.avail state=0x557e681411f0 s->prefetch_data=0 2021-02-11T09:43:29.258 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:29.254+0000 7f1293ac5080 10 couldn't find old data placement pools config, setting up new ones for the zone 2021-02-11T09:43:29.275 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:29.274+0000 7f1293ac5080 10 WARNING: store->list_zones() returned r=-2 2021-02-11T09:43:29.275 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:29.274+0000 7f1293ac5080 20 rados->read ofs=0 len=0 2021-02-11T09:43:29.275 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:29.274+0000 7f1293ac5080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:43:29.280 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:29.278+0000 7f1293ac5080 20 rados->read ofs=0 len=0 2021-02-11T09:43:29.281 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:29.278+0000 7f1293ac5080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:43:29.281 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:29.278+0000 7f1293ac5080 10 could not read realm id: (2) No such file or directory 2021-02-11T09:43:29.281 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:29.278+0000 7f1293ac5080 10 WARNING: failed to set zone as default, r=-22 2021-02-11T09:43:29.281 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:29.278+0000 7f1293ac5080 20 rados->read ofs=0 len=0 2021-02-11T09:43:29.281 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:29.278+0000 7f1293ac5080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:43:29.286 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:29.282+0000 7f1293ac5080 20 rados->read ofs=0 len=0 2021-02-11T09:43:29.287 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:29.286+0000 7f1293ac5080 20 rados_obj.operate() r=0 bl.length=873 2021-02-11T09:43:29.287 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:29.286+0000 7f1293ac5080 20 rados->read ofs=0 len=0 2021-02-11T09:43:29.288 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:29.286+0000 7f1293ac5080 20 rados_obj.operate() r=0 bl.length=358 2021-02-11T09:43:29.288 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:29.286+0000 7f1293ac5080 20 zonegroup default 2021-02-11T09:43:29.288 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:29.286+0000 7f1293ac5080 20 rados->read ofs=0 len=0 2021-02-11T09:43:29.289 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:29.286+0000 7f1293ac5080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:43:29.289 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:29.286+0000 7f1293ac5080 10 Cannot find current period zone using local zone 2021-02-11T09:43:29.289 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:29.286+0000 7f1293ac5080 10 Using default name default 2021-02-11T09:43:29.289 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:29.286+0000 7f1293ac5080 20 rados->read ofs=0 len=0 2021-02-11T09:43:29.289 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:29.286+0000 7f1293ac5080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T09:43:29.289 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:29.286+0000 7f1293ac5080 20 rados->read ofs=0 len=0 2021-02-11T09:43:29.290 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:29.286+0000 7f1293ac5080 20 rados_obj.operate() r=0 bl.length=873 2021-02-11T09:43:29.290 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:29.286+0000 7f1293ac5080 20 zone default found 2021-02-11T09:43:29.290 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:29.286+0000 7f1293ac5080 20 rados->read ofs=0 len=0 2021-02-11T09:43:31.259 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:31.257+0000 7f1293ac5080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:43:31.260 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:31.257+0000 7f1293ac5080 20 rados->read ofs=0 len=0 2021-02-11T09:43:31.260 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:31.261+0000 7f1293ac5080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:43:31.260 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:31.261+0000 7f1293ac5080 20 started sync module instance, tier type = 2021-02-11T09:43:31.260 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:31.261+0000 7f1293ac5080 20 started zone id=c9e1e86f-5612-404d-b586-60a6d48cc0bc (name=default) with tier type = 2021-02-11T09:43:33.300 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:33.297+0000 7f1293ac5080 20 add_watcher() i=0 2021-02-11T09:43:33.300 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:33.297+0000 7f1293ac5080 20 add_watcher() i=1 2021-02-11T09:43:33.300 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:33.297+0000 7f1293ac5080 20 add_watcher() i=2 2021-02-11T09:43:33.300 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:33.297+0000 7f1293ac5080 20 add_watcher() i=3 2021-02-11T09:43:33.300 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:33.297+0000 7f1293ac5080 20 add_watcher() i=4 2021-02-11T09:43:33.300 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:33.297+0000 7f1293ac5080 20 add_watcher() i=5 2021-02-11T09:43:33.300 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:33.297+0000 7f1293ac5080 20 add_watcher() i=6 2021-02-11T09:43:33.301 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:33.301+0000 7f1293ac5080 20 add_watcher() i=7 2021-02-11T09:43:33.301 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:33.301+0000 7f1293ac5080 2 all 8 watchers are set, enabling cache 2021-02-11T09:43:33.301 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:33.301+0000 7f1293ac5080 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(optional_yield):423: oid=data_loggenerations_metadata not found 2021-02-11T09:43:33.306 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:33.305+0000 7f1250fe1700 2 RGWDataChangesLog::ChangesRenewThread: start 2021-02-11T09:43:33.306 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:33.305+0000 7f1293ac5080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-02-11T09:43:33.306 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:33.305+0000 7f1293ac5080 20 check_secure_mon_conn(): mode 1 is insecure 2021-02-11T09:43:37.268 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:37.265+0000 7f1293ac5080 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2021-02-11T09:43:43.716 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:37.433+0000 7f11ab7fe700 20 reqs_thread_entry: start 2021-02-11T09:43:43.716 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:37.433+0000 7f1293ac5080 20 init_complete bucket index max shards: 11 2021-02-11T09:43:43.716 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:37.433+0000 7f11a97fa700 20 reqs_thread_entry: start 2021-02-11T09:43:43.716 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:37.433+0000 7f1293ac5080 10 rgw notify: Started notification manager with: 1 workers 2021-02-11T09:43:43.716 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:37.433+0000 7f11977fe700 20 rgw notify: INFO: next queues processing will happen at: Thu Feb 11 09:44:07 2021 2021-02-11T09:43:43.716 INFO:teuthology.orchestra.run.smithi094.stderr: 2021-02-11T09:43:43.716 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:37.433+0000 7f1196ffd700 20 reqs_thread_entry: start 2021-02-11T09:43:43.717 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:37.465+0000 7f1293ac5080 20 remove_watcher() i=0 2021-02-11T09:43:43.717 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:37.465+0000 7f1293ac5080 2 removed watcher, disabling cache 2021-02-11T09:43:43.717 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:37.469+0000 7f1293ac5080 20 remove_watcher() i=1 2021-02-11T09:43:43.717 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:37.473+0000 7f1293ac5080 20 remove_watcher() i=2 2021-02-11T09:43:43.717 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:37.477+0000 7f1293ac5080 20 remove_watcher() i=3 2021-02-11T09:43:43.717 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:37.477+0000 7f1293ac5080 20 remove_watcher() i=4 2021-02-11T09:43:43.717 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:37.481+0000 7f1293ac5080 20 remove_watcher() i=5 2021-02-11T09:43:43.717 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:37.481+0000 7f1293ac5080 20 remove_watcher() i=6 2021-02-11T09:43:43.717 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:37.485+0000 7f1293ac5080 20 remove_watcher() i=7 2021-02-11T09:43:43.717 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:37.485+0000 7f1250fe1700 2 RGWDataChangesLog::ChangesRenewThread: start 2021-02-11T09:43:43.717 INFO:teuthology.orchestra.run.smithi094.stdout:[] 2021-02-11T09:43:43.718 DEBUG:tasks.util.rgw: json result: [] 2021-02-11T09:43:43.718 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-02-11T09:43:43.718 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-02-11T09:43:43.718 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2021-02-11T09:43:43.800 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:43:43.801 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:43:43.814 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:43.813+0000 7f6c90474080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-02-11T09:43:43.814 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:43.813+0000 7f6c90474080 20 check_secure_mon_conn(): mode 1 is insecure 2021-02-11T09:43:43.822 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:43.821+0000 7f6bc5ffb700 20 reqs_thread_entry: start 2021-02-11T09:43:43.822 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:43.821+0000 7f6c90474080 20 rados->read ofs=0 len=0 2021-02-11T09:43:43.823 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:43.821+0000 7f6c90474080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T09:43:43.823 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:43.821+0000 7f6c90474080 20 rados->read ofs=0 len=0 2021-02-11T09:43:43.824 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:43.825+0000 7f6c90474080 20 rados_obj.operate() r=0 bl.length=873 2021-02-11T09:43:43.824 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:43.825+0000 7f6c90474080 20 rados->read ofs=0 len=0 2021-02-11T09:43:43.825 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:43.825+0000 7f6c90474080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:43:43.825 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:43.825+0000 7f6c90474080 20 rados->read ofs=0 len=0 2021-02-11T09:43:43.826 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:43.825+0000 7f6c90474080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T09:43:43.826 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:43.825+0000 7f6c90474080 20 rados->read ofs=0 len=0 2021-02-11T09:43:43.826 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:43.825+0000 7f6c90474080 20 rados_obj.operate() r=0 bl.length=358 2021-02-11T09:43:43.837 INFO:teuthology.orchestra.run.smithi094.stdout:{"id":"c9e1e86f-5612-404d-b586-60a6d48cc0bc","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0}}],"realm_id":"","notif_pool":"default.rgw.log:notif"} 2021-02-11T09:43:43.837 DEBUG:tasks.util.rgw: json result: {'id': 'c9e1e86f-5612-404d-b586-60a6d48cc0bc', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2021-02-11T09:43:43.837 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['user', 'list'] 2021-02-11T09:43:43.837 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'user', 'list'] 2021-02-11T09:43:43.837 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 user list 2021-02-11T09:43:43.916 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:43:43.917 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:43:43.927 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:43.925+0000 7ff9ab9ad080 20 rados->read ofs=0 len=0 2021-02-11T09:43:43.931 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:43.929+0000 7ff9ab9ad080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:43:43.931 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:43.929+0000 7ff9ab9ad080 20 rados->read ofs=0 len=0 2021-02-11T09:43:43.931 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:43.929+0000 7ff9ab9ad080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:43:43.931 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:43.929+0000 7ff9ab9ad080 20 rados->read ofs=0 len=0 2021-02-11T09:43:43.931 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:43.929+0000 7ff9ab9ad080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:43:43.931 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:43.929+0000 7ff9ab9ad080 20 rados->read ofs=0 len=0 2021-02-11T09:43:43.932 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:43.933+0000 7ff9ab9ad080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T09:43:43.955 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:43.953+0000 7ff9ab9ad080 20 RGWRados::pool_iterate: got zone_info.c9e1e86f-5612-404d-b586-60a6d48cc0bc 2021-02-11T09:43:43.956 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:43.953+0000 7ff9ab9ad080 20 RGWRados::pool_iterate: got zonegroup_info.919373db-f10b-4293-9203-9573f321166d 2021-02-11T09:43:43.956 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:43.953+0000 7ff9ab9ad080 20 RGWRados::pool_iterate: got zone_names.default 2021-02-11T09:43:43.956 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:43.953+0000 7ff9ab9ad080 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-02-11T09:43:43.956 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:43.953+0000 7ff9ab9ad080 20 rados->read ofs=0 len=0 2021-02-11T09:43:43.956 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:43.957+0000 7ff9ab9ad080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T09:43:43.956 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:43.957+0000 7ff9ab9ad080 20 rados->read ofs=0 len=0 2021-02-11T09:43:43.957 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:43.957+0000 7ff9ab9ad080 20 rados_obj.operate() r=0 bl.length=889 2021-02-11T09:43:43.957 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:43.957+0000 7ff9ab9ad080 20 rados->read ofs=0 len=0 2021-02-11T09:43:43.958 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:43.957+0000 7ff9ab9ad080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T09:43:43.958 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:43.957+0000 7ff9ab9ad080 20 rados->read ofs=0 len=0 2021-02-11T09:43:43.959 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:43.957+0000 7ff9ab9ad080 20 rados_obj.operate() r=0 bl.length=358 2021-02-11T09:43:43.959 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:43.957+0000 7ff9ab9ad080 20 rados->read ofs=0 len=0 2021-02-11T09:43:43.959 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:43.957+0000 7ff9ab9ad080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:43:43.959 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:43.957+0000 7ff9ab9ad080 10 cannot find current period zonegroup using local zonegroup 2021-02-11T09:43:43.960 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:43.957+0000 7ff9ab9ad080 20 rados->read ofs=0 len=0 2021-02-11T09:43:43.960 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:43.961+0000 7ff9ab9ad080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:43:43.960 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:43.961+0000 7ff9ab9ad080 20 rados->read ofs=0 len=0 2021-02-11T09:43:43.961 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:43.961+0000 7ff9ab9ad080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T09:43:43.961 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:43.961+0000 7ff9ab9ad080 20 rados->read ofs=0 len=0 2021-02-11T09:43:43.961 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:43.961+0000 7ff9ab9ad080 20 rados_obj.operate() r=0 bl.length=358 2021-02-11T09:43:43.962 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:43.961+0000 7ff9ab9ad080 20 zonegroup default 2021-02-11T09:43:43.962 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:43.961+0000 7ff9ab9ad080 20 rados->read ofs=0 len=0 2021-02-11T09:43:43.962 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:43.961+0000 7ff9ab9ad080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:43:43.963 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:43.961+0000 7ff9ab9ad080 10 Cannot find current period zone using local zone 2021-02-11T09:43:43.963 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:43.961+0000 7ff9ab9ad080 20 rados->read ofs=0 len=0 2021-02-11T09:43:43.963 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:43.961+0000 7ff9ab9ad080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:43:43.964 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:43.961+0000 7ff9ab9ad080 20 rados->read ofs=0 len=0 2021-02-11T09:43:43.964 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:43.965+0000 7ff9ab9ad080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T09:43:43.965 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:43.965+0000 7ff9ab9ad080 20 rados->read ofs=0 len=0 2021-02-11T09:43:43.965 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:43.965+0000 7ff9ab9ad080 20 rados_obj.operate() r=0 bl.length=889 2021-02-11T09:43:43.966 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:43.965+0000 7ff9ab9ad080 20 zone default found 2021-02-11T09:43:43.966 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:43.965+0000 7ff9ab9ad080 20 rados->read ofs=0 len=0 2021-02-11T09:43:43.966 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:43.965+0000 7ff9ab9ad080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:43:43.966 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:43.965+0000 7ff9ab9ad080 20 rados->read ofs=0 len=0 2021-02-11T09:43:43.967 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:43.965+0000 7ff9ab9ad080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:43:43.967 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:43.965+0000 7ff9ab9ad080 20 started sync module instance, tier type = 2021-02-11T09:43:43.967 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:43.965+0000 7ff9ab9ad080 20 started zone id=c9e1e86f-5612-404d-b586-60a6d48cc0bc (name=default) with tier type = 2021-02-11T09:43:43.986 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:43.985+0000 7ff9ab9ad080 20 add_watcher() i=0 2021-02-11T09:43:43.986 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:43.985+0000 7ff9ab9ad080 20 add_watcher() i=1 2021-02-11T09:43:43.986 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:43.985+0000 7ff9ab9ad080 20 add_watcher() i=2 2021-02-11T09:43:43.986 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:43.985+0000 7ff9ab9ad080 20 add_watcher() i=3 2021-02-11T09:43:43.986 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:43.985+0000 7ff9ab9ad080 20 add_watcher() i=4 2021-02-11T09:43:43.987 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:43.985+0000 7ff9ab9ad080 20 add_watcher() i=5 2021-02-11T09:43:43.987 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:43.985+0000 7ff9ab9ad080 20 add_watcher() i=6 2021-02-11T09:43:43.988 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:43.989+0000 7ff9ab9ad080 20 add_watcher() i=7 2021-02-11T09:43:43.988 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:43.989+0000 7ff9ab9ad080 2 all 8 watchers are set, enabling cache 2021-02-11T09:43:43.991 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:43.989+0000 7ff968fe1700 2 RGWDataChangesLog::ChangesRenewThread: start 2021-02-11T09:43:43.991 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:43.989+0000 7ff9ab9ad080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-02-11T09:43:43.991 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:43.989+0000 7ff9ab9ad080 20 check_secure_mon_conn(): mode 1 is insecure 2021-02-11T09:43:44.078 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:44.077+0000 7ff8d77fe700 20 reqs_thread_entry: start 2021-02-11T09:43:44.079 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:44.077+0000 7ff8d57fa700 20 reqs_thread_entry: start 2021-02-11T09:43:44.079 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:44.077+0000 7ff9ab9ad080 20 init_complete bucket index max shards: 11 2021-02-11T09:43:44.079 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:44.077+0000 7ff9ab9ad080 10 rgw notify: Started notification manager with: 1 workers 2021-02-11T09:43:44.079 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:44.077+0000 7ff8c2ffd700 20 reqs_thread_entry: start 2021-02-11T09:43:44.079 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:44.077+0000 7ff8c37fe700 20 rgw notify: INFO: next queues processing will happen at: Thu Feb 11 09:44:14 2021 2021-02-11T09:43:44.080 INFO:teuthology.orchestra.run.smithi094.stderr: 2021-02-11T09:43:44.102 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:44.101+0000 7ff9ab9ad080 20 remove_watcher() i=0 2021-02-11T09:43:44.103 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:44.101+0000 7ff9ab9ad080 2 removed watcher, disabling cache 2021-02-11T09:43:44.105 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:44.105+0000 7ff9ab9ad080 20 remove_watcher() i=1 2021-02-11T09:43:44.107 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:44.105+0000 7ff9ab9ad080 20 remove_watcher() i=2 2021-02-11T09:43:44.110 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:44.109+0000 7ff9ab9ad080 20 remove_watcher() i=3 2021-02-11T09:43:44.112 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:44.113+0000 7ff9ab9ad080 20 remove_watcher() i=4 2021-02-11T09:43:44.115 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:44.113+0000 7ff9ab9ad080 20 remove_watcher() i=5 2021-02-11T09:43:44.118 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:44.117+0000 7ff9ab9ad080 20 remove_watcher() i=6 2021-02-11T09:43:44.120 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:44.121+0000 7ff9ab9ad080 20 remove_watcher() i=7 2021-02-11T09:43:44.122 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:44.121+0000 7ff968fe1700 2 RGWDataChangesLog::ChangesRenewThread: start 2021-02-11T09:43:44.131 INFO:teuthology.orchestra.run.smithi094.stdout:[] 2021-02-11T09:43:44.131 DEBUG:tasks.util.rgw: json result: [] 2021-02-11T09:43:44.131 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-02-11T09:43:44.131 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-02-11T09:43:44.131 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2021-02-11T09:43:44.212 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:43:44.213 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:43:44.223 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:44.221+0000 7fe03a57f080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-02-11T09:43:44.223 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:44.221+0000 7fe03a57f080 20 check_secure_mon_conn(): mode 1 is insecure 2021-02-11T09:43:44.229 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:44.229+0000 7fdf6ffff700 20 reqs_thread_entry: start 2021-02-11T09:43:44.229 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:44.229+0000 7fe03a57f080 20 rados->read ofs=0 len=0 2021-02-11T09:43:44.231 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:44.229+0000 7fe03a57f080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T09:43:44.231 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:44.229+0000 7fe03a57f080 20 rados->read ofs=0 len=0 2021-02-11T09:43:44.232 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:44.233+0000 7fe03a57f080 20 rados_obj.operate() r=0 bl.length=889 2021-02-11T09:43:44.232 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:44.233+0000 7fe03a57f080 20 rados->read ofs=0 len=0 2021-02-11T09:43:44.232 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:44.233+0000 7fe03a57f080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:43:44.232 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:44.233+0000 7fe03a57f080 20 rados->read ofs=0 len=0 2021-02-11T09:43:44.233 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:44.233+0000 7fe03a57f080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T09:43:44.233 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:44.233+0000 7fe03a57f080 20 rados->read ofs=0 len=0 2021-02-11T09:43:44.233 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:44.233+0000 7fe03a57f080 20 rados_obj.operate() r=0 bl.length=358 2021-02-11T09:43:44.242 INFO:teuthology.orchestra.run.smithi094.stdout:{"id":"c9e1e86f-5612-404d-b586-60a6d48cc0bc","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0}}],"realm_id":"","notif_pool":"default.rgw.log:notif"} 2021-02-11T09:43:44.242 DEBUG:tasks.util.rgw: json result: {'id': 'c9e1e86f-5612-404d-b586-60a6d48cc0bc', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2021-02-11T09:43:44.242 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['user', 'list'] 2021-02-11T09:43:44.243 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'user', 'list'] 2021-02-11T09:43:44.243 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 user list 2021-02-11T09:43:44.278 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:43:44.279 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:43:44.291 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:44.290+0000 7f472be19080 20 rados->read ofs=0 len=0 2021-02-11T09:43:46.281 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:46.278+0000 7f472be19080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:43:46.281 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:46.278+0000 7f472be19080 20 rados->read ofs=0 len=0 2021-02-11T09:43:46.282 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:46.282+0000 7f472be19080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:43:46.282 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:46.282+0000 7f472be19080 20 rados->read ofs=0 len=0 2021-02-11T09:43:46.282 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:46.282+0000 7f472be19080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:43:46.282 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:46.282+0000 7f472be19080 20 rados->read ofs=0 len=0 2021-02-11T09:43:46.283 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:46.282+0000 7f472be19080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:43:46.304 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:46.302+0000 7f472be19080 20 rados->read ofs=0 len=0 2021-02-11T09:43:46.304 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:46.302+0000 7f472be19080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:43:46.304 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:46.302+0000 7f472be19080 20 rados->read ofs=0 len=0 2021-02-11T09:43:46.304 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:46.302+0000 7f472be19080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:43:46.304 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:46.302+0000 7f472be19080 20 rados->read ofs=0 len=0 2021-02-11T09:43:46.305 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:46.302+0000 7f472be19080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:43:46.305 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:46.302+0000 7f472be19080 10 cannot find current period zonegroup using local zonegroup 2021-02-11T09:43:46.305 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:46.302+0000 7f472be19080 20 rados->read ofs=0 len=0 2021-02-11T09:43:46.305 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:46.306+0000 7f472be19080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:43:46.305 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:46.306+0000 7f472be19080 20 rados->read ofs=0 len=0 2021-02-11T09:43:46.306 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:46.306+0000 7f472be19080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:43:46.306 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:46.306+0000 7f472be19080 10 Creating default zonegroup 2021-02-11T09:43:46.306 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:46.306+0000 7f472be19080 20 get_system_obj_state: rctx=0x7fff61019300 obj=:.pools.avail state=0x5596882d21f0 s->prefetch_data=0 2021-02-11T09:43:46.306 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:46.306+0000 7f472be19080 10 couldn't find old data placement pools config, setting up new ones for the zone 2021-02-11T09:43:46.322 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:46.322+0000 7f472be19080 10 WARNING: store->list_zones() returned r=-2 2021-02-11T09:43:46.322 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:46.322+0000 7f472be19080 20 rados->read ofs=0 len=0 2021-02-11T09:43:46.323 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:46.322+0000 7f472be19080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:43:46.327 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:46.326+0000 7f472be19080 20 rados->read ofs=0 len=0 2021-02-11T09:43:46.328 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:46.326+0000 7f472be19080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:43:46.328 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:46.326+0000 7f472be19080 10 could not read realm id: (2) No such file or directory 2021-02-11T09:43:46.328 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:46.326+0000 7f472be19080 10 WARNING: failed to set zone as default, r=-22 2021-02-11T09:43:46.328 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:46.326+0000 7f472be19080 20 rados->read ofs=0 len=0 2021-02-11T09:43:46.328 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:46.326+0000 7f472be19080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:43:46.334 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:46.334+0000 7f472be19080 20 rados->read ofs=0 len=0 2021-02-11T09:43:46.334 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:46.334+0000 7f472be19080 20 rados_obj.operate() r=0 bl.length=873 2021-02-11T09:43:46.334 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:46.334+0000 7f472be19080 20 rados->read ofs=0 len=0 2021-02-11T09:43:46.335 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:46.334+0000 7f472be19080 20 rados_obj.operate() r=0 bl.length=358 2021-02-11T09:43:46.335 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:46.334+0000 7f472be19080 20 zonegroup default 2021-02-11T09:43:46.335 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:46.334+0000 7f472be19080 20 rados->read ofs=0 len=0 2021-02-11T09:43:46.336 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:46.334+0000 7f472be19080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:43:46.336 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:46.334+0000 7f472be19080 10 Cannot find current period zone using local zone 2021-02-11T09:43:46.336 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:46.334+0000 7f472be19080 10 Using default name default 2021-02-11T09:43:46.336 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:46.334+0000 7f472be19080 20 rados->read ofs=0 len=0 2021-02-11T09:43:46.337 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:46.334+0000 7f472be19080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T09:43:46.337 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:46.334+0000 7f472be19080 20 rados->read ofs=0 len=0 2021-02-11T09:43:46.337 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:46.338+0000 7f472be19080 20 rados_obj.operate() r=0 bl.length=873 2021-02-11T09:43:46.338 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:46.338+0000 7f472be19080 20 zone default found 2021-02-11T09:43:46.338 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:46.338+0000 7f472be19080 20 rados->read ofs=0 len=0 2021-02-11T09:43:48.296 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:48.294+0000 7f472be19080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:43:48.296 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:48.294+0000 7f472be19080 20 rados->read ofs=0 len=0 2021-02-11T09:43:48.296 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:48.294+0000 7f472be19080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:43:48.296 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:48.294+0000 7f472be19080 20 started sync module instance, tier type = 2021-02-11T09:43:48.296 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:48.294+0000 7f472be19080 20 started zone id=4a5df491-fb02-4496-9506-36e7c1f9c57c (name=default) with tier type = 2021-02-11T09:43:50.389 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:50.386+0000 7f472be19080 20 add_watcher() i=0 2021-02-11T09:43:50.389 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:50.386+0000 7f472be19080 20 add_watcher() i=1 2021-02-11T09:43:50.389 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:50.386+0000 7f472be19080 20 add_watcher() i=2 2021-02-11T09:43:50.389 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:50.386+0000 7f472be19080 20 add_watcher() i=3 2021-02-11T09:43:50.389 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:50.386+0000 7f472be19080 20 add_watcher() i=4 2021-02-11T09:43:50.389 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:50.386+0000 7f472be19080 20 add_watcher() i=5 2021-02-11T09:43:50.389 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:50.386+0000 7f472be19080 20 add_watcher() i=6 2021-02-11T09:43:50.391 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:50.390+0000 7f472be19080 20 add_watcher() i=7 2021-02-11T09:43:50.391 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:50.390+0000 7f472be19080 2 all 8 watchers are set, enabling cache 2021-02-11T09:43:50.391 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:50.390+0000 7f472be19080 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(optional_yield):423: oid=data_loggenerations_metadata not found 2021-02-11T09:43:50.400 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:50.398+0000 7f4663fff700 2 RGWDataChangesLog::ChangesRenewThread: start 2021-02-11T09:43:50.400 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:50.398+0000 7f472be19080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-02-11T09:43:50.400 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:50.398+0000 7f472be19080 20 check_secure_mon_conn(): mode 1 is insecure 2021-02-11T09:43:54.343 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.342+0000 7f472be19080 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2021-02-11T09:43:54.470 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.470+0000 7f4656ffd700 20 reqs_thread_entry: start 2021-02-11T09:43:54.470 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.470+0000 7f472be19080 20 init_complete bucket index max shards: 11 2021-02-11T09:43:54.470 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.470+0000 7f4654ff9700 20 reqs_thread_entry: start 2021-02-11T09:43:54.471 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.470+0000 7f472be19080 10 rgw notify: Started notification manager with: 1 workers 2021-02-11T09:43:54.471 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.470+0000 7f463affd700 20 rgw notify: INFO: next queues processing will happen at: Thu Feb 11 09:44:24 2021 2021-02-11T09:43:54.471 INFO:teuthology.orchestra.run.smithi098.stderr: 2021-02-11T09:43:54.471 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.470+0000 7f463a7fc700 20 reqs_thread_entry: start 2021-02-11T09:43:54.502 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.502+0000 7f472be19080 20 remove_watcher() i=0 2021-02-11T09:43:54.502 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.502+0000 7f472be19080 2 removed watcher, disabling cache 2021-02-11T09:43:54.505 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.506+0000 7f472be19080 20 remove_watcher() i=1 2021-02-11T09:43:54.508 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.506+0000 7f472be19080 20 remove_watcher() i=2 2021-02-11T09:43:54.511 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.510+0000 7f472be19080 20 remove_watcher() i=3 2021-02-11T09:43:54.514 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.514+0000 7f472be19080 20 remove_watcher() i=4 2021-02-11T09:43:54.517 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.514+0000 7f472be19080 20 remove_watcher() i=5 2021-02-11T09:43:54.522 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.522+0000 7f472be19080 20 remove_watcher() i=6 2021-02-11T09:43:54.524 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.522+0000 7f472be19080 20 remove_watcher() i=7 2021-02-11T09:43:54.526 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.526+0000 7f4663fff700 2 RGWDataChangesLog::ChangesRenewThread: start 2021-02-11T09:43:54.536 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2021-02-11T09:43:54.536 DEBUG:tasks.util.rgw: json result: [] 2021-02-11T09:43:54.536 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-02-11T09:43:54.536 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-02-11T09:43:54.536 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2021-02-11T09:43:54.618 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:43:54.618 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:43:54.630 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.630+0000 7f1bd7d8a080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-02-11T09:43:54.630 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.630+0000 7f1bd7d8a080 20 check_secure_mon_conn(): mode 1 is insecure 2021-02-11T09:43:54.640 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.638+0000 7f1b2d7fa700 20 reqs_thread_entry: start 2021-02-11T09:43:54.640 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.638+0000 7f1bd7d8a080 20 rados->read ofs=0 len=0 2021-02-11T09:43:54.641 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.642+0000 7f1bd7d8a080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T09:43:54.641 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.642+0000 7f1bd7d8a080 20 rados->read ofs=0 len=0 2021-02-11T09:43:54.643 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.642+0000 7f1bd7d8a080 20 rados_obj.operate() r=0 bl.length=873 2021-02-11T09:43:54.643 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.642+0000 7f1bd7d8a080 20 rados->read ofs=0 len=0 2021-02-11T09:43:54.644 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.642+0000 7f1bd7d8a080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:43:54.644 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.642+0000 7f1bd7d8a080 20 rados->read ofs=0 len=0 2021-02-11T09:43:54.645 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.646+0000 7f1bd7d8a080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T09:43:54.645 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.646+0000 7f1bd7d8a080 20 rados->read ofs=0 len=0 2021-02-11T09:43:54.645 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.646+0000 7f1bd7d8a080 20 rados_obj.operate() r=0 bl.length=358 2021-02-11T09:43:54.656 INFO:teuthology.orchestra.run.smithi098.stdout:{"id":"4a5df491-fb02-4496-9506-36e7c1f9c57c","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0}}],"realm_id":"","notif_pool":"default.rgw.log:notif"} 2021-02-11T09:43:54.656 DEBUG:tasks.util.rgw: json result: {'id': '4a5df491-fb02-4496-9506-36e7c1f9c57c', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2021-02-11T09:43:54.656 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['user', 'list'] 2021-02-11T09:43:54.656 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'user', 'list'] 2021-02-11T09:43:54.656 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 user list 2021-02-11T09:43:54.733 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:43:54.734 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:43:54.742 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.742+0000 7fe30dbdf080 20 rados->read ofs=0 len=0 2021-02-11T09:43:54.745 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.746+0000 7fe30dbdf080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:43:54.746 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.746+0000 7fe30dbdf080 20 rados->read ofs=0 len=0 2021-02-11T09:43:54.746 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.746+0000 7fe30dbdf080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:43:54.746 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.746+0000 7fe30dbdf080 20 rados->read ofs=0 len=0 2021-02-11T09:43:54.747 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.746+0000 7fe30dbdf080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:43:54.747 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.746+0000 7fe30dbdf080 20 rados->read ofs=0 len=0 2021-02-11T09:43:54.747 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.746+0000 7fe30dbdf080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T09:43:54.771 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.770+0000 7fe30dbdf080 20 RGWRados::pool_iterate: got zone_info.4a5df491-fb02-4496-9506-36e7c1f9c57c 2021-02-11T09:43:54.772 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.770+0000 7fe30dbdf080 20 RGWRados::pool_iterate: got zone_names.default 2021-02-11T09:43:54.772 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.770+0000 7fe30dbdf080 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-02-11T09:43:54.772 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.770+0000 7fe30dbdf080 20 RGWRados::pool_iterate: got zonegroup_info.d434d675-1d9a-4fb4-aeb1-facfb0b3a726 2021-02-11T09:43:54.772 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.770+0000 7fe30dbdf080 20 rados->read ofs=0 len=0 2021-02-11T09:43:54.772 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.770+0000 7fe30dbdf080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T09:43:54.772 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.770+0000 7fe30dbdf080 20 rados->read ofs=0 len=0 2021-02-11T09:43:54.773 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.774+0000 7fe30dbdf080 20 rados_obj.operate() r=0 bl.length=889 2021-02-11T09:43:54.773 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.774+0000 7fe30dbdf080 20 rados->read ofs=0 len=0 2021-02-11T09:43:54.774 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.774+0000 7fe30dbdf080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T09:43:54.774 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.774+0000 7fe30dbdf080 20 rados->read ofs=0 len=0 2021-02-11T09:43:54.775 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.774+0000 7fe30dbdf080 20 rados_obj.operate() r=0 bl.length=358 2021-02-11T09:43:54.775 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.774+0000 7fe30dbdf080 20 rados->read ofs=0 len=0 2021-02-11T09:43:54.775 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.774+0000 7fe30dbdf080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:43:54.775 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.774+0000 7fe30dbdf080 10 cannot find current period zonegroup using local zonegroup 2021-02-11T09:43:54.776 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.774+0000 7fe30dbdf080 20 rados->read ofs=0 len=0 2021-02-11T09:43:54.776 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.774+0000 7fe30dbdf080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:43:54.776 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.774+0000 7fe30dbdf080 20 rados->read ofs=0 len=0 2021-02-11T09:43:54.777 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.778+0000 7fe30dbdf080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T09:43:54.777 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.778+0000 7fe30dbdf080 20 rados->read ofs=0 len=0 2021-02-11T09:43:54.778 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.778+0000 7fe30dbdf080 20 rados_obj.operate() r=0 bl.length=358 2021-02-11T09:43:54.778 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.778+0000 7fe30dbdf080 20 zonegroup default 2021-02-11T09:43:54.778 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.778+0000 7fe30dbdf080 20 rados->read ofs=0 len=0 2021-02-11T09:43:54.778 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.778+0000 7fe30dbdf080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:43:54.778 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.778+0000 7fe30dbdf080 10 Cannot find current period zone using local zone 2021-02-11T09:43:54.778 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.778+0000 7fe30dbdf080 20 rados->read ofs=0 len=0 2021-02-11T09:43:54.779 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.778+0000 7fe30dbdf080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:43:54.779 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.778+0000 7fe30dbdf080 20 rados->read ofs=0 len=0 2021-02-11T09:43:54.779 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.778+0000 7fe30dbdf080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T09:43:54.780 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.778+0000 7fe30dbdf080 20 rados->read ofs=0 len=0 2021-02-11T09:43:54.781 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.778+0000 7fe30dbdf080 20 rados_obj.operate() r=0 bl.length=889 2021-02-11T09:43:54.781 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.782+0000 7fe30dbdf080 20 zone default found 2021-02-11T09:43:54.781 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.782+0000 7fe30dbdf080 20 rados->read ofs=0 len=0 2021-02-11T09:43:54.781 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.782+0000 7fe30dbdf080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:43:54.782 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.782+0000 7fe30dbdf080 20 rados->read ofs=0 len=0 2021-02-11T09:43:54.782 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.782+0000 7fe30dbdf080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:43:54.782 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.782+0000 7fe30dbdf080 20 started sync module instance, tier type = 2021-02-11T09:43:54.782 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.782+0000 7fe30dbdf080 20 started zone id=4a5df491-fb02-4496-9506-36e7c1f9c57c (name=default) with tier type = 2021-02-11T09:43:54.812 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.810+0000 7fe30dbdf080 20 add_watcher() i=0 2021-02-11T09:43:54.812 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.810+0000 7fe30dbdf080 20 add_watcher() i=1 2021-02-11T09:43:54.812 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.810+0000 7fe30dbdf080 20 add_watcher() i=2 2021-02-11T09:43:54.812 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.810+0000 7fe30dbdf080 20 add_watcher() i=3 2021-02-11T09:43:54.813 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.810+0000 7fe30dbdf080 20 add_watcher() i=4 2021-02-11T09:43:54.813 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.810+0000 7fe30dbdf080 20 add_watcher() i=5 2021-02-11T09:43:54.813 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.810+0000 7fe30dbdf080 20 add_watcher() i=6 2021-02-11T09:43:54.814 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.814+0000 7fe30dbdf080 20 add_watcher() i=7 2021-02-11T09:43:54.814 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.814+0000 7fe30dbdf080 2 all 8 watchers are set, enabling cache 2021-02-11T09:43:54.818 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.818+0000 7fe2ca7e4700 2 RGWDataChangesLog::ChangesRenewThread: start 2021-02-11T09:43:54.818 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.818+0000 7fe30dbdf080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-02-11T09:43:54.818 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.818+0000 7fe30dbdf080 20 check_secure_mon_conn(): mode 1 is insecure 2021-02-11T09:43:54.915 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.914+0000 7fe2397fa700 20 reqs_thread_entry: start 2021-02-11T09:43:54.915 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.914+0000 7fe232ffd700 20 reqs_thread_entry: start 2021-02-11T09:43:54.915 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.914+0000 7fe30dbdf080 20 init_complete bucket index max shards: 11 2021-02-11T09:43:54.916 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.914+0000 7fe30dbdf080 10 rgw notify: Started notification manager with: 1 workers 2021-02-11T09:43:54.916 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.914+0000 7fe2317fa700 20 rgw notify: INFO: next queues processing will happen at: Thu Feb 11 09:44:25 2021 2021-02-11T09:43:54.916 INFO:teuthology.orchestra.run.smithi098.stderr: 2021-02-11T09:43:54.916 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.914+0000 7fe230ff9700 20 reqs_thread_entry: start 2021-02-11T09:43:54.945 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.946+0000 7fe30dbdf080 20 remove_watcher() i=0 2021-02-11T09:43:54.945 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.946+0000 7fe30dbdf080 2 removed watcher, disabling cache 2021-02-11T09:43:54.948 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.946+0000 7fe30dbdf080 20 remove_watcher() i=1 2021-02-11T09:43:54.951 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.950+0000 7fe30dbdf080 20 remove_watcher() i=2 2021-02-11T09:43:54.954 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.954+0000 7fe30dbdf080 20 remove_watcher() i=3 2021-02-11T09:43:54.957 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.958+0000 7fe30dbdf080 20 remove_watcher() i=4 2021-02-11T09:43:54.960 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.958+0000 7fe30dbdf080 20 remove_watcher() i=5 2021-02-11T09:43:54.962 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.962+0000 7fe30dbdf080 20 remove_watcher() i=6 2021-02-11T09:43:54.964 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.962+0000 7fe30dbdf080 20 remove_watcher() i=7 2021-02-11T09:43:54.966 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:54.966+0000 7fe2ca7e4700 2 RGWDataChangesLog::ChangesRenewThread: start 2021-02-11T09:43:54.975 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2021-02-11T09:43:54.976 DEBUG:tasks.util.rgw: json result: [] 2021-02-11T09:43:54.976 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-02-11T09:43:54.976 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-02-11T09:43:54.976 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2021-02-11T09:43:55.057 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:43:55.058 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:43:55.067 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:55.066+0000 7f87f8a57080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-02-11T09:43:55.067 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:55.066+0000 7f87f8a57080 20 check_secure_mon_conn(): mode 1 is insecure 2021-02-11T09:43:55.074 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:55.074+0000 7f872dffb700 20 reqs_thread_entry: start 2021-02-11T09:43:55.074 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:55.074+0000 7f87f8a57080 20 rados->read ofs=0 len=0 2021-02-11T09:43:55.077 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:55.074+0000 7f87f8a57080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T09:43:55.077 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:55.078+0000 7f87f8a57080 20 rados->read ofs=0 len=0 2021-02-11T09:43:55.078 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:55.078+0000 7f87f8a57080 20 rados_obj.operate() r=0 bl.length=889 2021-02-11T09:43:55.079 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:55.078+0000 7f87f8a57080 20 rados->read ofs=0 len=0 2021-02-11T09:43:55.079 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:55.078+0000 7f87f8a57080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T09:43:55.079 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:55.078+0000 7f87f8a57080 20 rados->read ofs=0 len=0 2021-02-11T09:43:55.080 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:55.078+0000 7f87f8a57080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T09:43:55.080 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:55.078+0000 7f87f8a57080 20 rados->read ofs=0 len=0 2021-02-11T09:43:55.080 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:43:55.078+0000 7f87f8a57080 20 rados_obj.operate() r=0 bl.length=358 2021-02-11T09:43:55.089 INFO:teuthology.orchestra.run.smithi098.stdout:{"id":"4a5df491-fb02-4496-9506-36e7c1f9c57c","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0}}],"realm_id":"","notif_pool":"default.rgw.log:notif"} 2021-02-11T09:43:55.090 DEBUG:tasks.util.rgw: json result: {'id': '4a5df491-fb02-4496-9506-36e7c1f9c57c', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2021-02-11T09:43:55.090 INFO:tasks.rgw:Starting rgw... 2021-02-11T09:43:55.090 INFO:tasks.rgw:rgw c1.client.0 config is {'port': 8000} 2021-02-11T09:43:55.090 INFO:tasks.rgw:Using beast as radosgw frontend 2021-02-11T09:43:55.090 INFO:tasks.rgw.c1.client.0:Restarting daemon 2021-02-11T09:43:55.090 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2021-02-11T09:43:55.093 INFO:tasks.rgw.c1.client.0:Started 2021-02-11T09:43:55.093 INFO:tasks.rgw:rgw c1.client.1 config is {'port': 8001} 2021-02-11T09:43:55.093 INFO:tasks.rgw:Using beast as radosgw frontend 2021-02-11T09:43:55.093 INFO:tasks.rgw.c1.client.1:Restarting daemon 2021-02-11T09:43:55.093 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2021-02-11T09:43:55.095 INFO:tasks.rgw.c1.client.1:Started 2021-02-11T09:43:55.095 INFO:tasks.rgw:rgw c2.client.0 config is {'port': 8000} 2021-02-11T09:43:55.096 INFO:tasks.rgw:Using beast as radosgw frontend 2021-02-11T09:43:55.096 INFO:tasks.rgw.c2.client.0:Restarting daemon 2021-02-11T09:43:55.096 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2021-02-11T09:43:55.134 INFO:tasks.rgw.c2.client.0:Started 2021-02-11T09:43:55.134 INFO:tasks.rgw:rgw c2.client.1 config is {'port': 8001} 2021-02-11T09:43:55.134 INFO:tasks.rgw:Using beast as radosgw frontend 2021-02-11T09:43:55.134 INFO:tasks.rgw.c2.client.1:Restarting daemon 2021-02-11T09:43:55.135 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2021-02-11T09:43:55.137 INFO:tasks.rgw.c2.client.1:Started 2021-02-11T09:43:55.137 INFO:tasks.rgw:Polling c1.client.0 until it starts accepting connections on http://smithi094.front.sepia.ceph.com:8000/ 2021-02-11T09:43:55.137 DEBUG:teuthology.orchestra.run.smithi094:> curl http://smithi094.front.sepia.ceph.com:8000/ 2021-02-11T09:43:55.170 INFO:teuthology.orchestra.run.smithi094.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-11T09:43:55.170 INFO:teuthology.orchestra.run.smithi094.stderr: Dload Upload Total Spent Left Speed 2021-02-11T09:43:55.174 INFO:teuthology.orchestra.run.smithi094.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi094.front.sepia.ceph.com port 8000: Connection refused 2021-02-11T09:43:55.175 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-02-11T09:43:56.175 DEBUG:teuthology.orchestra.run.smithi094:> curl http://smithi094.front.sepia.ceph.com:8000/ 2021-02-11T09:43:56.193 INFO:teuthology.orchestra.run.smithi094.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-11T09:43:56.193 INFO:teuthology.orchestra.run.smithi094.stderr: Dload Upload Total Spent Left Speed 2021-02-11T09:43:56.201 INFO:teuthology.orchestra.run.smithi094.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 26750 0 --:--:-- --:--:-- --:--:-- 26750 2021-02-11T09:43:56.202 INFO:teuthology.orchestra.run.smithi094.stdout:anonymous 2021-02-11T09:43:56.202 INFO:tasks.rgw:Polling c1.client.1 until it starts accepting connections on http://smithi094.front.sepia.ceph.com:8001/ 2021-02-11T09:43:56.203 DEBUG:teuthology.orchestra.run.smithi094:> curl http://smithi094.front.sepia.ceph.com:8001/ 2021-02-11T09:43:56.258 INFO:teuthology.orchestra.run.smithi094.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-11T09:43:56.258 INFO:teuthology.orchestra.run.smithi094.stderr: Dload Upload Total Spent Left Speed 2021-02-11T09:43:56.267 INFO:teuthology.orchestra.run.smithi094.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 26750 0 --:--:-- --:--:-- --:--:-- 26750 2021-02-11T09:43:56.268 INFO:teuthology.orchestra.run.smithi094.stdout:anonymous 2021-02-11T09:43:56.268 INFO:tasks.rgw:Polling c2.client.0 until it starts accepting connections on http://smithi098.front.sepia.ceph.com:8000/ 2021-02-11T09:43:56.268 DEBUG:teuthology.orchestra.run.smithi098:> curl http://smithi098.front.sepia.ceph.com:8000/ 2021-02-11T09:43:56.301 INFO:teuthology.orchestra.run.smithi098.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-11T09:43:56.301 INFO:teuthology.orchestra.run.smithi098.stderr: Dload Upload Total Spent Left Speed 2021-02-11T09:43:56.308 INFO:teuthology.orchestra.run.smithi098.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 16461 0 --:--:-- --:--:-- --:--:-- 35666 2021-02-11T09:43:56.308 INFO:teuthology.orchestra.run.smithi098.stdout:anonymous 2021-02-11T09:43:56.309 INFO:tasks.rgw:Polling c2.client.1 until it starts accepting connections on http://smithi098.front.sepia.ceph.com:8001/ 2021-02-11T09:43:56.309 DEBUG:teuthology.orchestra.run.smithi098:> curl http://smithi098.front.sepia.ceph.com:8001/ 2021-02-11T09:43:56.365 INFO:teuthology.orchestra.run.smithi098.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-11T09:43:56.365 INFO:teuthology.orchestra.run.smithi098.stderr: Dload Upload Total Spent Left Speed 2021-02-11T09:43:56.372 INFO:teuthology.orchestra.run.smithi098.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 30571 0 --:--:-- --:--:-- --:--:-- 30571 2021-02-11T09:43:56.373 INFO:teuthology.orchestra.run.smithi098.stdout:anonymous 2021-02-11T09:43:56.373 INFO:teuthology.run_tasks:Running task rgw-multisite... 2021-02-11T09:43:56.854 INFO:tasks.rgw_multisite:creating realm.. 2021-02-11T09:43:56.855 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:43:56.855 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:43:56.855 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 realm create --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T09:43:56.890 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:43:56.890 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:43:56.939 INFO:teuthology.orchestra.run.smithi094.stdout:{"id":"7e807acb-79b4-4a09-ba75-06bbedad4839","name":"test-realm","current_period":"09ea9628-da02-4170-9251-46ca95429c49","epoch":1} 2021-02-11T09:43:56.939 INFO:tasks.rgw_multisite:creating master zonegroup.. 2021-02-11T09:43:56.939 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:43:56.939 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:43:56.940 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup set --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T09:43:57.019 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:43:57.020 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:43:57.043 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:43:57.041+0000 7f558b247080 -1 old format 2021-02-11T09:43:57.059 INFO:teuthology.orchestra.run.smithi094.stdout:{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi094.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"","zones":[],"placement_targets":[],"default_placement":"","realm_id":"7e807acb-79b4-4a09-ba75-06bbedad4839","sync_policy":{"groups":[]}} 2021-02-11T09:43:57.059 INFO:tasks.rgw_multisite:creating master zone.. 2021-02-11T09:43:57.059 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', '7B4L28ZSM4TSLFS6', '--secret', 'eZwNt56BcNJm7h6ZlwjNkhL3ytjGBlGP', '--endpoints', 'http://smithi094.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:43:57.059 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', '7B4L28ZSM4TSLFS6', '--secret', 'eZwNt56BcNJm7h6ZlwjNkhL3ytjGBlGP', '--endpoints', 'http://smithi094.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:43:57.060 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --master --access-key 7B4L28ZSM4TSLFS6 --secret eZwNt56BcNJm7h6ZlwjNkhL3ytjGBlGP --endpoints http://smithi094.front.sepia.ceph.com:8000 --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T09:43:57.144 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:43:57.144 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:43:57.216 INFO:teuthology.orchestra.run.smithi094.stdout:{"id":"dd308248-8ee8-49be-b928-07c3de7d7aa2","name":"test-zone1","domain_root":"test-zone1.rgw.meta:root","control_pool":"test-zone1.rgw.control","gc_pool":"test-zone1.rgw.log:gc","lc_pool":"test-zone1.rgw.log:lc","log_pool":"test-zone1.rgw.log","intent_log_pool":"test-zone1.rgw.log:intent","usage_log_pool":"test-zone1.rgw.log:usage","roles_pool":"test-zone1.rgw.meta:roles","reshard_pool":"test-zone1.rgw.log:reshard","user_keys_pool":"test-zone1.rgw.meta:users.keys","user_email_pool":"test-zone1.rgw.meta:users.email","user_swift_pool":"test-zone1.rgw.meta:users.swift","user_uid_pool":"test-zone1.rgw.meta:users.uid","otp_pool":"test-zone1.rgw.otp","system_key":{"access_key":"7B4L28ZSM4TSLFS6","secret_key":"eZwNt56BcNJm7h6ZlwjNkhL3ytjGBlGP"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone1.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone1.rgw.buckets.data"}},"data_extra_pool":"test-zone1.rgw.buckets.non-ec","index_type":0}}],"realm_id":"7e807acb-79b4-4a09-ba75-06bbedad4839","notif_pool":"test-zone1.rgw.log:notif"} 2021-02-11T09:43:57.217 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph osd pool create test-zone1.rgw.buckets.data 64 64 --cluster c1 2021-02-11T09:43:58.242 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'test-zone1.rgw.buckets.data' created 2021-02-11T09:43:58.258 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph osd pool application enable test-zone1.rgw.buckets.data rgw --cluster c1 2021-02-11T09:43:59.266 INFO:teuthology.orchestra.run.smithi094.stderr:enabled application 'rgw' on pool 'test-zone1.rgw.buckets.data' 2021-02-11T09:43:59.292 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:43:59.292 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:43:59.292 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T09:43:59.329 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:43:59.329 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:43:59.354 INFO:teuthology.orchestra.run.smithi094.stdout:{"id":"dd308248-8ee8-49be-b928-07c3de7d7aa2","name":"test-zone1","domain_root":"test-zone1.rgw.meta:root","control_pool":"test-zone1.rgw.control","gc_pool":"test-zone1.rgw.log:gc","lc_pool":"test-zone1.rgw.log:lc","log_pool":"test-zone1.rgw.log","intent_log_pool":"test-zone1.rgw.log:intent","usage_log_pool":"test-zone1.rgw.log:usage","roles_pool":"test-zone1.rgw.meta:roles","reshard_pool":"test-zone1.rgw.log:reshard","user_keys_pool":"test-zone1.rgw.meta:users.keys","user_email_pool":"test-zone1.rgw.meta:users.email","user_swift_pool":"test-zone1.rgw.meta:users.swift","user_uid_pool":"test-zone1.rgw.meta:users.uid","otp_pool":"test-zone1.rgw.otp","system_key":{"access_key":"7B4L28ZSM4TSLFS6","secret_key":"eZwNt56BcNJm7h6ZlwjNkhL3ytjGBlGP"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone1.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone1.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone1.rgw.buckets.non-ec","index_type":0}}],"realm_id":"","notif_pool":"test-zone1.rgw.log:notif"} 2021-02-11T09:43:59.354 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:43:59.354 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:43:59.354 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T09:43:59.434 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:43:59.434 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:44:07.493 INFO:teuthology.orchestra.run.smithi094.stdout:{"id":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","epoch":1,"predecessor_uuid":"09ea9628-da02-4170-9251-46ca95429c49","sync_status":[],"period_map":{"id":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi094.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"dd308248-8ee8-49be-b928-07c3de7d7aa2","zones":[{"id":"dd308248-8ee8-49be-b928-07c3de7d7aa2","name":"test-zone1","endpoints":["http://smithi094.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"false","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"7e807acb-79b4-4a09-ba75-06bbedad4839","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"dd308248-8ee8-49be-b928-07c3de7d7aa2","val":207523491}]},"master_zonegroup":"test-zonegroup","master_zone":"dd308248-8ee8-49be-b928-07c3de7d7aa2","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"7e807acb-79b4-4a09-ba75-06bbedad4839","realm_name":"test-realm","realm_epoch":2} 2021-02-11T09:44:07.493 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2021-02-11T09:44:07.493 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T09:44:07.536 INFO:tasks.rgw.c1.client.0.smithi094.stdout:2021-02-11T09:44:07.537+0000 7f51d8f83700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground (PID: 16896) UID: 0 2021-02-11T09:44:07.536 INFO:tasks.rgw.c1.client.0.smithi094.stdout:2021-02-11T09:44:07.537+0000 7f51f0ade0c0 -1 shutting down 2021-02-11T09:44:12.234 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2021-02-11T09:44:13.595 INFO:tasks.rgw.c1.client.0:Stopped 2021-02-11T09:44:13.596 INFO:tasks.rgw.c1.client.0:Restarting daemon 2021-02-11T09:44:13.596 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2021-02-11T09:44:13.598 INFO:tasks.rgw.c1.client.0:Started 2021-02-11T09:44:13.599 DEBUG:teuthology.orchestra.run.smithi094:> curl http://smithi094.front.sepia.ceph.com:8000 2021-02-11T09:44:13.615 INFO:teuthology.orchestra.run.smithi094.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-11T09:44:13.615 INFO:teuthology.orchestra.run.smithi094.stderr: Dload Upload Total Spent Left Speed 2021-02-11T09:44:13.619 INFO:teuthology.orchestra.run.smithi094.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi094.front.sepia.ceph.com port 8000: Connection refused 2021-02-11T09:44:13.620 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-02-11T09:44:14.621 DEBUG:teuthology.orchestra.run.smithi094:> curl http://smithi094.front.sepia.ceph.com:8000 2021-02-11T09:44:14.640 INFO:teuthology.orchestra.run.smithi094.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-11T09:44:14.640 INFO:teuthology.orchestra.run.smithi094.stderr: Dload Upload Total Spent Left Speed 2021-02-11T09:44:14.643 INFO:teuthology.orchestra.run.smithi094.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 53500 0 --:--:-- --:--:-- --:--:-- 71333 2021-02-11T09:44:14.644 INFO:teuthology.orchestra.run.smithi094.stdout:anonymous 2021-02-11T09:44:14.644 INFO:tasks.rgw_multisite:creating admin user.. 2021-02-11T09:44:14.645 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', '7B4L28ZSM4TSLFS6', '--secret', 'eZwNt56BcNJm7h6ZlwjNkhL3ytjGBlGP', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:44:14.645 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', '7B4L28ZSM4TSLFS6', '--secret', 'eZwNt56BcNJm7h6ZlwjNkhL3ytjGBlGP', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:44:14.645 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid realm-admin --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Realm Admin' --system --access-key 7B4L28ZSM4TSLFS6 --secret eZwNt56BcNJm7h6ZlwjNkhL3ytjGBlGP --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T09:44:14.732 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:44:14.732 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:44:14.905 INFO:teuthology.orchestra.run.smithi094.stdout:{"user_id":"realm-admin","display_name":"Realm Admin","email":"","suspended":0,"max_buckets":1000,"subusers":[],"keys":[{"user":"realm-admin","access_key":"7B4L28ZSM4TSLFS6","secret_key":"eZwNt56BcNJm7h6ZlwjNkhL3ytjGBlGP"}],"swift_keys":[],"caps":[],"op_mask":"read, write, delete","system":"true","default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[]} 2021-02-11T09:44:14.936 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2021-02-11T09:44:14.936 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi094.front.sepia.ceph.com:8000', '--access-key', '7B4L28ZSM4TSLFS6', '--secret', 'eZwNt56BcNJm7h6ZlwjNkhL3ytjGBlGP', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:44:14.936 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi094.front.sepia.ceph.com:8000', '--access-key', '7B4L28ZSM4TSLFS6', '--secret', 'eZwNt56BcNJm7h6ZlwjNkhL3ytjGBlGP', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:44:14.936 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi094.front.sepia.ceph.com:8000 --access-key 7B4L28ZSM4TSLFS6 --secret eZwNt56BcNJm7h6ZlwjNkhL3ytjGBlGP --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T09:44:14.979 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:44:14.979 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:44:15.104 INFO:teuthology.orchestra.run.smithi098.stdout:{"id":"7e807acb-79b4-4a09-ba75-06bbedad4839","name":"test-realm","current_period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","epoch":2} 2021-02-11T09:44:15.104 INFO:tasks.rgw_multisite:creating zone.. 2021-02-11T09:44:15.105 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', '7B4L28ZSM4TSLFS6', '--secret', 'eZwNt56BcNJm7h6ZlwjNkhL3ytjGBlGP', '--endpoints', 'http://smithi098.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:44:15.105 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', '7B4L28ZSM4TSLFS6', '--secret', 'eZwNt56BcNJm7h6ZlwjNkhL3ytjGBlGP', '--endpoints', 'http://smithi098.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:44:15.105 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --access-key 7B4L28ZSM4TSLFS6 --secret eZwNt56BcNJm7h6ZlwjNkhL3ytjGBlGP --endpoints http://smithi098.front.sepia.ceph.com:8000 --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T09:44:15.190 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:44:15.190 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:44:15.249 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:44:15.250+0000 7f6a9d949080 0 failed reading obj info from .rgw.root:zone_info.dd308248-8ee8-49be-b928-07c3de7d7aa2: (2) No such file or directory 2021-02-11T09:44:15.250 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:44:15.250+0000 7f6a9d949080 0 WARNING: could not read zone params for zone id=dd308248-8ee8-49be-b928-07c3de7d7aa2 name=test-zone1 2021-02-11T09:44:15.261 INFO:teuthology.orchestra.run.smithi098.stdout:{"id":"63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"7B4L28ZSM4TSLFS6","secret_key":"eZwNt56BcNJm7h6ZlwjNkhL3ytjGBlGP"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"7e807acb-79b4-4a09-ba75-06bbedad4839","notif_pool":"test-zone2.rgw.log:notif"} 2021-02-11T09:44:15.261 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph osd pool create test-zone2.rgw.buckets.data 64 64 --cluster c2 2021-02-11T09:44:16.293 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'test-zone2.rgw.buckets.data' created 2021-02-11T09:44:16.315 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph osd pool application enable test-zone2.rgw.buckets.data rgw --cluster c2 2021-02-11T09:44:17.316 INFO:teuthology.orchestra.run.smithi098.stderr:enabled application 'rgw' on pool 'test-zone2.rgw.buckets.data' 2021-02-11T09:44:17.343 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:44:17.343 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:44:17.343 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T09:44:17.383 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:44:17.384 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:44:17.410 INFO:teuthology.orchestra.run.smithi098.stdout:{"id":"63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"7B4L28ZSM4TSLFS6","secret_key":"eZwNt56BcNJm7h6ZlwjNkhL3ytjGBlGP"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"","notif_pool":"test-zone2.rgw.log:notif"} 2021-02-11T09:44:17.411 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:44:17.411 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:44:17.411 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T09:44:17.495 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:44:17.495 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:44:18.590 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2021-02-11T09:44:25.530 INFO:teuthology.orchestra.run.smithi098.stderr:Sending period to new master zone dd308248-8ee8-49be-b928-07c3de7d7aa2 2021-02-11T09:44:25.630 INFO:teuthology.orchestra.run.smithi098.stdout:{"id":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","epoch":2,"predecessor_uuid":"09ea9628-da02-4170-9251-46ca95429c49","sync_status":[],"period_map":{"id":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi094.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"dd308248-8ee8-49be-b928-07c3de7d7aa2","zones":[{"id":"63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2","name":"test-zone2","endpoints":["http://smithi098.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"dd308248-8ee8-49be-b928-07c3de7d7aa2","name":"test-zone1","endpoints":["http://smithi094.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"7e807acb-79b4-4a09-ba75-06bbedad4839","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2","val":1552258706},{"key":"dd308248-8ee8-49be-b928-07c3de7d7aa2","val":207523491}]},"master_zonegroup":"test-zonegroup","master_zone":"dd308248-8ee8-49be-b928-07c3de7d7aa2","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"7e807acb-79b4-4a09-ba75-06bbedad4839","realm_name":"test-realm","realm_epoch":2} 2021-02-11T09:44:25.631 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2021-02-11T09:44:25.631 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T09:44:25.673 INFO:tasks.rgw.c2.client.0.smithi098.stdout:2021-02-11T09:44:25.674+0000 7f7450aa1700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground (PID: 16937) UID: 0 2021-02-11T09:44:25.673 INFO:tasks.rgw.c2.client.0.smithi098.stdout:2021-02-11T09:44:25.674+0000 7f74685fc0c0 -1 shutting down 2021-02-11T09:44:30.004 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2021-02-11T09:44:31.733 INFO:tasks.rgw.c2.client.0:Stopped 2021-02-11T09:44:31.733 INFO:tasks.rgw.c2.client.0:Restarting daemon 2021-02-11T09:44:31.733 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2021-02-11T09:44:31.736 INFO:tasks.rgw.c2.client.0:Started 2021-02-11T09:44:31.736 DEBUG:teuthology.orchestra.run.smithi098:> curl http://smithi098.front.sepia.ceph.com:8000 2021-02-11T09:44:31.752 INFO:teuthology.orchestra.run.smithi098.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-11T09:44:31.752 INFO:teuthology.orchestra.run.smithi098.stderr: Dload Upload Total Spent Left Speed 2021-02-11T09:44:31.756 INFO:teuthology.orchestra.run.smithi098.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi098.front.sepia.ceph.com port 8000: Connection refused 2021-02-11T09:44:31.756 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-02-11T09:44:32.758 DEBUG:teuthology.orchestra.run.smithi098:> curl http://smithi098.front.sepia.ceph.com:8000 2021-02-11T09:44:32.772 INFO:teuthology.orchestra.run.smithi098.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-11T09:44:32.772 INFO:teuthology.orchestra.run.smithi098.stderr: Dload Upload Total Spent Left Speed 2021-02-11T09:44:32.779 INFO:teuthology.orchestra.run.smithi098.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 30571 0 --:--:-- --:--:-- --:--:-- 30571 2021-02-11T09:44:32.780 INFO:teuthology.orchestra.run.smithi098.stdout:anonymous 2021-02-11T09:44:32.780 INFO:tasks.rgw_multisite:creating zone.. 2021-02-11T09:44:32.781 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', '7B4L28ZSM4TSLFS6', '--secret', 'eZwNt56BcNJm7h6ZlwjNkhL3ytjGBlGP', '--endpoints', 'http://smithi094.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:44:32.781 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', '7B4L28ZSM4TSLFS6', '--secret', 'eZwNt56BcNJm7h6ZlwjNkhL3ytjGBlGP', '--endpoints', 'http://smithi094.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:44:32.781 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key 7B4L28ZSM4TSLFS6 --secret eZwNt56BcNJm7h6ZlwjNkhL3ytjGBlGP --endpoints http://smithi094.front.sepia.ceph.com:8001 --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T09:44:32.825 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:44:32.825 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:44:32.885 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:44:32.885+0000 7f5a1a45e080 0 failed reading obj info from .rgw.root:zone_info.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2: (2) No such file or directory 2021-02-11T09:44:32.885 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:44:32.885+0000 7f5a1a45e080 0 WARNING: could not read zone params for zone id=63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2 name=test-zone2 2021-02-11T09:44:32.897 INFO:teuthology.orchestra.run.smithi094.stdout:{"id":"92d097d8-38ae-4839-9777-3b30db2b2f60","name":"test-zone3","domain_root":"test-zone3.rgw.meta:root","control_pool":"test-zone3.rgw.control","gc_pool":"test-zone3.rgw.log:gc","lc_pool":"test-zone3.rgw.log:lc","log_pool":"test-zone3.rgw.log","intent_log_pool":"test-zone3.rgw.log:intent","usage_log_pool":"test-zone3.rgw.log:usage","roles_pool":"test-zone3.rgw.meta:roles","reshard_pool":"test-zone3.rgw.log:reshard","user_keys_pool":"test-zone3.rgw.meta:users.keys","user_email_pool":"test-zone3.rgw.meta:users.email","user_swift_pool":"test-zone3.rgw.meta:users.swift","user_uid_pool":"test-zone3.rgw.meta:users.uid","otp_pool":"test-zone3.rgw.otp","system_key":{"access_key":"7B4L28ZSM4TSLFS6","secret_key":"eZwNt56BcNJm7h6ZlwjNkhL3ytjGBlGP"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone3.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone3.rgw.buckets.data"}},"data_extra_pool":"test-zone3.rgw.buckets.non-ec","index_type":0}}],"realm_id":"7e807acb-79b4-4a09-ba75-06bbedad4839","notif_pool":"test-zone3.rgw.log:notif"} 2021-02-11T09:44:32.897 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph osd pool create test-zone3.rgw.buckets.data 64 64 --cluster c1 2021-02-11T09:44:33.259 INFO:teuthology.orchestra.run.smithi094.stderr:pool 'test-zone3.rgw.buckets.data' created 2021-02-11T09:44:33.274 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph osd pool application enable test-zone3.rgw.buckets.data rgw --cluster c1 2021-02-11T09:44:34.289 INFO:teuthology.orchestra.run.smithi094.stderr:enabled application 'rgw' on pool 'test-zone3.rgw.buckets.data' 2021-02-11T09:44:34.316 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:44:34.316 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:44:34.316 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T09:44:34.373 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:44:34.373 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:44:34.394 INFO:teuthology.orchestra.run.smithi094.stdout:{"id":"92d097d8-38ae-4839-9777-3b30db2b2f60","name":"test-zone3","domain_root":"test-zone3.rgw.meta:root","control_pool":"test-zone3.rgw.control","gc_pool":"test-zone3.rgw.log:gc","lc_pool":"test-zone3.rgw.log:lc","log_pool":"test-zone3.rgw.log","intent_log_pool":"test-zone3.rgw.log:intent","usage_log_pool":"test-zone3.rgw.log:usage","roles_pool":"test-zone3.rgw.meta:roles","reshard_pool":"test-zone3.rgw.log:reshard","user_keys_pool":"test-zone3.rgw.meta:users.keys","user_email_pool":"test-zone3.rgw.meta:users.email","user_swift_pool":"test-zone3.rgw.meta:users.swift","user_uid_pool":"test-zone3.rgw.meta:users.uid","otp_pool":"test-zone3.rgw.otp","system_key":{"access_key":"7B4L28ZSM4TSLFS6","secret_key":"eZwNt56BcNJm7h6ZlwjNkhL3ytjGBlGP"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone3.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone3.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone3.rgw.buckets.non-ec","index_type":0}}],"realm_id":"","notif_pool":"test-zone3.rgw.log:notif"} 2021-02-11T09:44:34.394 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:44:34.395 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:44:34.395 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T09:44:34.427 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:44:34.428 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:44:35.708 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2021-02-11T09:44:42.485 INFO:teuthology.orchestra.run.smithi094.stderr:Sending period to new master zone dd308248-8ee8-49be-b928-07c3de7d7aa2 2021-02-11T09:44:42.573 INFO:teuthology.orchestra.run.smithi094.stdout:{"id":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","epoch":3,"predecessor_uuid":"09ea9628-da02-4170-9251-46ca95429c49","sync_status":[],"period_map":{"id":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi094.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"dd308248-8ee8-49be-b928-07c3de7d7aa2","zones":[{"id":"63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2","name":"test-zone2","endpoints":["http://smithi098.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"92d097d8-38ae-4839-9777-3b30db2b2f60","name":"test-zone3","endpoints":["http://smithi094.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"dd308248-8ee8-49be-b928-07c3de7d7aa2","name":"test-zone1","endpoints":["http://smithi094.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"7e807acb-79b4-4a09-ba75-06bbedad4839","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2","val":1552258706},{"key":"92d097d8-38ae-4839-9777-3b30db2b2f60","val":2008974102},{"key":"dd308248-8ee8-49be-b928-07c3de7d7aa2","val":207523491}]},"master_zonegroup":"test-zonegroup","master_zone":"dd308248-8ee8-49be-b928-07c3de7d7aa2","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"7e807acb-79b4-4a09-ba75-06bbedad4839","realm_name":"test-realm","realm_epoch":2} 2021-02-11T09:44:42.573 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2021-02-11T09:44:42.573 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T09:44:42.615 INFO:tasks.rgw.c1.client.1.smithi094.stdout:2021-02-11T09:44:42.617+0000 7fad037fe700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground (PID: 16898) UID: 0 2021-02-11T09:44:42.616 INFO:tasks.rgw.c1.client.1.smithi094.stdout:2021-02-11T09:44:42.617+0000 7fad1ef7f0c0 -1 shutting down 2021-02-11T09:44:47.015 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2021-02-11T09:44:48.674 INFO:tasks.rgw.c1.client.1:Stopped 2021-02-11T09:44:48.675 INFO:tasks.rgw.c1.client.1:Restarting daemon 2021-02-11T09:44:48.675 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2021-02-11T09:44:48.678 INFO:tasks.rgw.c1.client.1:Started 2021-02-11T09:44:48.678 DEBUG:teuthology.orchestra.run.smithi094:> curl http://smithi094.front.sepia.ceph.com:8001 2021-02-11T09:44:48.694 INFO:teuthology.orchestra.run.smithi094.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-11T09:44:48.694 INFO:teuthology.orchestra.run.smithi094.stderr: Dload Upload Total Spent Left Speed 2021-02-11T09:44:48.698 INFO:teuthology.orchestra.run.smithi094.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi094.front.sepia.ceph.com port 8001: Connection refused 2021-02-11T09:44:48.699 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-02-11T09:44:49.699 DEBUG:teuthology.orchestra.run.smithi094:> curl http://smithi094.front.sepia.ceph.com:8001 2021-02-11T09:44:49.714 INFO:teuthology.orchestra.run.smithi094.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-11T09:44:49.714 INFO:teuthology.orchestra.run.smithi094.stderr: Dload Upload Total Spent Left Speed 2021-02-11T09:44:49.723 INFO:teuthology.orchestra.run.smithi094.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 26750 0 --:--:-- --:--:-- --:--:-- 26750 2021-02-11T09:44:49.724 INFO:teuthology.orchestra.run.smithi094.stdout:anonymous 2021-02-11T09:44:49.724 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2021-02-11T09:44:49.724 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi094.front.sepia.ceph.com:8000', '--access-key', '7B4L28ZSM4TSLFS6', '--secret', 'eZwNt56BcNJm7h6ZlwjNkhL3ytjGBlGP', '--url', 'http://smithi094.front.sepia.ceph.com:8000', '--access-key', '7B4L28ZSM4TSLFS6', '--secret', 'eZwNt56BcNJm7h6ZlwjNkhL3ytjGBlGP', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:44:49.724 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi094.front.sepia.ceph.com:8000', '--access-key', '7B4L28ZSM4TSLFS6', '--secret', 'eZwNt56BcNJm7h6ZlwjNkhL3ytjGBlGP', '--url', 'http://smithi094.front.sepia.ceph.com:8000', '--access-key', '7B4L28ZSM4TSLFS6', '--secret', 'eZwNt56BcNJm7h6ZlwjNkhL3ytjGBlGP', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:44:49.724 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi094.front.sepia.ceph.com:8000 --access-key 7B4L28ZSM4TSLFS6 --secret eZwNt56BcNJm7h6ZlwjNkhL3ytjGBlGP --url http://smithi094.front.sepia.ceph.com:8000 --access-key 7B4L28ZSM4TSLFS6 --secret eZwNt56BcNJm7h6ZlwjNkhL3ytjGBlGP --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T09:44:49.775 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:44:49.775 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:44:49.821 INFO:tasks.rgw.c2.client.0.smithi098.stdout:2021-02-11T09:44:49.822+0000 7f29bbe0f700 -1 rgw realm watcher: RGWRealmWatcher::handle_error oid=realms.7e807acb-79b4-4a09-ba75-06bbedad4839.control err=-107 2021-02-11T09:44:49.845 INFO:teuthology.orchestra.run.smithi098.stdout:{"id":"7e807acb-79b4-4a09-ba75-06bbedad4839","name":"test-realm","current_period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","epoch":2} 2021-02-11T09:44:49.846 INFO:tasks.rgw_multisite:creating zone.. 2021-02-11T09:44:49.846 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', '7B4L28ZSM4TSLFS6', '--secret', 'eZwNt56BcNJm7h6ZlwjNkhL3ytjGBlGP', '--endpoints', 'http://smithi098.front.sepia.ceph.com:8001', '--tier-type', 'pubsub', '--sync-from-all=0', '--sync-from', 'test-zone1', '--tier-config', 'start_with_full_sync=false,event_retention_days=7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:44:49.846 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', '7B4L28ZSM4TSLFS6', '--secret', 'eZwNt56BcNJm7h6ZlwjNkhL3ytjGBlGP', '--endpoints', 'http://smithi098.front.sepia.ceph.com:8001', '--tier-type', 'pubsub', '--sync-from-all=0', '--sync-from', 'test-zone1', '--tier-config', 'start_with_full_sync=false,event_retention_days=7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:44:49.846 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key 7B4L28ZSM4TSLFS6 --secret eZwNt56BcNJm7h6ZlwjNkhL3ytjGBlGP --endpoints http://smithi098.front.sepia.ceph.com:8001 --tier-type pubsub --sync-from-all=0 --sync-from test-zone1 --tier-config start_with_full_sync=false,event_retention_days=7 --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T09:44:49.895 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:44:49.896 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:44:49.965 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:44:49.966+0000 7f3c5b90b080 0 failed reading obj info from .rgw.root:zone_info.92d097d8-38ae-4839-9777-3b30db2b2f60: (2) No such file or directory 2021-02-11T09:44:49.966 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:44:49.966+0000 7f3c5b90b080 0 WARNING: could not read zone params for zone id=92d097d8-38ae-4839-9777-3b30db2b2f60 name=test-zone3 2021-02-11T09:44:49.966 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:44:49.966+0000 7f3c5b90b080 0 failed reading obj info from .rgw.root:zone_info.dd308248-8ee8-49be-b928-07c3de7d7aa2: (2) No such file or directory 2021-02-11T09:44:49.966 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:44:49.966+0000 7f3c5b90b080 0 WARNING: could not read zone params for zone id=dd308248-8ee8-49be-b928-07c3de7d7aa2 name=test-zone1 2021-02-11T09:44:49.973 INFO:teuthology.orchestra.run.smithi098.stdout:{"id":"45265506-b5d1-4710-9a9e-3e67bee0e451","name":"test-zone4","domain_root":"test-zone4.rgw.meta:root","control_pool":"test-zone4.rgw.control","gc_pool":"test-zone4.rgw.log:gc","lc_pool":"test-zone4.rgw.log:lc","log_pool":"test-zone4.rgw.log","intent_log_pool":"test-zone4.rgw.log:intent","usage_log_pool":"test-zone4.rgw.log:usage","roles_pool":"test-zone4.rgw.meta:roles","reshard_pool":"test-zone4.rgw.log:reshard","user_keys_pool":"test-zone4.rgw.meta:users.keys","user_email_pool":"test-zone4.rgw.meta:users.email","user_swift_pool":"test-zone4.rgw.meta:users.swift","user_uid_pool":"test-zone4.rgw.meta:users.uid","otp_pool":"test-zone4.rgw.otp","system_key":{"access_key":"7B4L28ZSM4TSLFS6","secret_key":"eZwNt56BcNJm7h6ZlwjNkhL3ytjGBlGP"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone4.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone4.rgw.buckets.data"}},"data_extra_pool":"test-zone4.rgw.buckets.non-ec","index_type":0}}],"tier_config":{"event_retention_days":7,"start_with_full_sync":false},"realm_id":"7e807acb-79b4-4a09-ba75-06bbedad4839","notif_pool":"test-zone4.rgw.log:notif"} 2021-02-11T09:44:49.973 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph osd pool create test-zone4.rgw.buckets.data 64 64 --cluster c2 2021-02-11T09:44:51.293 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'test-zone4.rgw.buckets.data' created 2021-02-11T09:44:51.312 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph osd pool application enable test-zone4.rgw.buckets.data rgw --cluster c2 2021-02-11T09:44:52.312 INFO:teuthology.orchestra.run.smithi098.stderr:enabled application 'rgw' on pool 'test-zone4.rgw.buckets.data' 2021-02-11T09:44:52.347 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:44:52.347 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:44:52.347 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T09:44:52.388 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:44:52.389 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:44:52.412 INFO:teuthology.orchestra.run.smithi098.stdout:{"id":"45265506-b5d1-4710-9a9e-3e67bee0e451","name":"test-zone4","domain_root":"test-zone4.rgw.meta:root","control_pool":"test-zone4.rgw.control","gc_pool":"test-zone4.rgw.log:gc","lc_pool":"test-zone4.rgw.log:lc","log_pool":"test-zone4.rgw.log","intent_log_pool":"test-zone4.rgw.log:intent","usage_log_pool":"test-zone4.rgw.log:usage","roles_pool":"test-zone4.rgw.meta:roles","reshard_pool":"test-zone4.rgw.log:reshard","user_keys_pool":"test-zone4.rgw.meta:users.keys","user_email_pool":"test-zone4.rgw.meta:users.email","user_swift_pool":"test-zone4.rgw.meta:users.swift","user_uid_pool":"test-zone4.rgw.meta:users.uid","otp_pool":"test-zone4.rgw.otp","system_key":{"access_key":"7B4L28ZSM4TSLFS6","secret_key":"eZwNt56BcNJm7h6ZlwjNkhL3ytjGBlGP"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone4.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone4.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone4.rgw.buckets.non-ec","index_type":0}}],"tier_config":{"event_retention_days":7,"start_with_full_sync":false},"realm_id":"","notif_pool":"test-zone4.rgw.log:notif"} 2021-02-11T09:44:52.413 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:44:52.413 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:44:52.413 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T09:44:52.491 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:44:52.491 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:44:52.719 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2021-02-11T09:45:00.538 INFO:teuthology.orchestra.run.smithi098.stderr:Sending period to new master zone dd308248-8ee8-49be-b928-07c3de7d7aa2 2021-02-11T09:45:00.629 INFO:teuthology.orchestra.run.smithi098.stdout:{"id":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","epoch":4,"predecessor_uuid":"09ea9628-da02-4170-9251-46ca95429c49","sync_status":[],"period_map":{"id":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi094.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"dd308248-8ee8-49be-b928-07c3de7d7aa2","zones":[{"id":"45265506-b5d1-4710-9a9e-3e67bee0e451","name":"test-zone4","endpoints":["http://smithi098.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2","name":"test-zone2","endpoints":["http://smithi098.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"92d097d8-38ae-4839-9777-3b30db2b2f60","name":"test-zone3","endpoints":["http://smithi094.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"dd308248-8ee8-49be-b928-07c3de7d7aa2","name":"test-zone1","endpoints":["http://smithi094.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"7e807acb-79b4-4a09-ba75-06bbedad4839","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"45265506-b5d1-4710-9a9e-3e67bee0e451","val":240795374},{"key":"63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2","val":1552258706},{"key":"92d097d8-38ae-4839-9777-3b30db2b2f60","val":2008974102},{"key":"dd308248-8ee8-49be-b928-07c3de7d7aa2","val":207523491}]},"master_zonegroup":"test-zonegroup","master_zone":"dd308248-8ee8-49be-b928-07c3de7d7aa2","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"7e807acb-79b4-4a09-ba75-06bbedad4839","realm_name":"test-realm","realm_epoch":2} 2021-02-11T09:45:00.630 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2021-02-11T09:45:00.630 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T09:45:00.672 INFO:tasks.rgw.c2.client.1.smithi098.stdout:2021-02-11T09:45:00.674+0000 7f7097fff700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground (PID: 16938) UID: 0 2021-02-11T09:45:00.672 INFO:tasks.rgw.c2.client.1.smithi098.stdout:2021-02-11T09:45:00.674+0000 7f70b41880c0 -1 shutting down 2021-02-11T09:45:04.134 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2021-02-11T09:45:06.732 INFO:tasks.rgw.c2.client.1:Stopped 2021-02-11T09:45:06.732 INFO:tasks.rgw.c2.client.1:Restarting daemon 2021-02-11T09:45:06.732 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2021-02-11T09:45:06.735 INFO:tasks.rgw.c2.client.1:Started 2021-02-11T09:45:06.735 DEBUG:teuthology.orchestra.run.smithi098:> curl http://smithi098.front.sepia.ceph.com:8001 2021-02-11T09:45:06.751 INFO:teuthology.orchestra.run.smithi098.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-11T09:45:06.751 INFO:teuthology.orchestra.run.smithi098.stderr: Dload Upload Total Spent Left Speed 2021-02-11T09:45:06.755 INFO:teuthology.orchestra.run.smithi098.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi098.front.sepia.ceph.com port 8001: Connection refused 2021-02-11T09:45:06.756 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-02-11T09:45:07.756 DEBUG:teuthology.orchestra.run.smithi098:> curl http://smithi098.front.sepia.ceph.com:8001 2021-02-11T09:45:07.771 INFO:teuthology.orchestra.run.smithi098.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-11T09:45:07.771 INFO:teuthology.orchestra.run.smithi098.stderr: Dload Upload Total Spent Left Speed 2021-02-11T09:45:07.775 INFO:teuthology.orchestra.run.smithi098.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 134 100 134 0 0 33500 0 --:--:-- --:--:-- --:--:-- 33500 2021-02-11T09:45:07.776 INFO:teuthology.orchestra.run.smithi098.stdout:{"Code":"MethodNotAllowed","RequestId":"tx000000000000000000001-006024fca3-10e4-test-zone4","HostId":"10e4-test-zone4-test-zonegroup"} 2021-02-11T09:45:07.776 INFO:tasks.rgw_multisite:rgw multisite configuration completed 2021-02-11T09:45:07.777 INFO:teuthology.run_tasks:Running task rgw-multisite-tests... 2021-02-11T09:45:07.866 INFO:tasks.rgw_multisite_tests:creating test user.. 2021-02-11T09:45:07.866 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:45:07.866 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:45:07.866 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid rgw-multisite-test-user --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Multisite Test User' --gen-access-key --gen-secret --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T09:45:07.913 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:45:07.913 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:45:08.100 INFO:teuthology.orchestra.run.smithi094.stdout:{"user_id":"rgw-multisite-test-user","display_name":"Multisite Test User","email":"","suspended":0,"max_buckets":1000,"subusers":[],"keys":[{"user":"rgw-multisite-test-user","access_key":"T1B3OMGK8KX68Y6U60J2","secret_key":"jxqFUxMIzANGo6pfV7qc5OgagifvpgduVW25dkC1"}],"swift_keys":[],"caps":[],"op_mask":"read, write, delete","default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[]} 2021-02-11T09:45:08.130 INFO:rgw_multi.tests:meta checkpoint 2021-02-11T09:45:08.130 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:45:08.130 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:45:08.131 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:45:08.169 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:45:08.169 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:45:08.343 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1 2021-02-11T09:45:08.374 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613036708.096437_52.1","last_update":"2021-02-11T09:45:08.096437Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T09:45:08.374 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T09:45:08.375 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:45:08.375 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:45:08.375 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:45:08.412 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:45:08.412 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:45:08.635 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:45:08.636 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613036708.096437_52.1 target= 2021-02-11T09:45:09.840 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 has been restored 2021-02-11T09:45:13.637 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:45:13.637 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:45:13.638 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:45:13.679 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:45:13.679 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:45:13.905 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:45:13.905 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613036708.096437_52.1 target= 2021-02-11T09:45:18.906 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:45:18.907 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:45:18.907 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:45:18.947 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:45:18.947 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:45:19.167 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:45:19.167 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613036708.096437_52.1 target= 2021-02-11T09:45:24.168 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:45:31.523 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:45:31.524 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:45:31.565 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:45:31.565 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:45:31.785 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:45:31.786 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613036708.096437_52.1 target= 2021-02-11T09:45:36.787 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:45:36.787 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:45:36.787 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:45:36.830 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:45:36.830 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:45:37.043 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:45:37.043 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613036708.096437_52.1 target= 2021-02-11T09:45:42.044 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:45:42.044 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:45:42.044 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:45:42.086 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:45:42.086 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:45:42.299 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:45:42.299 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613036708.096437_52.1 target= 2021-02-11T09:45:47.300 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:45:47.301 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:45:47.301 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:45:47.337 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:45:47.337 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:45:47.544 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:45:47.544 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613036708.096437_52.1 target= 2021-02-11T09:45:52.545 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:45:52.545 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:45:52.545 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:45:52.587 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:45:52.587 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:45:52.807 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:45:52.808 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613036708.096437_52.1 target= 2021-02-11T09:45:57.808 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:45:57.808 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:45:57.809 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:45:57.850 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:45:57.850 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:45:58.078 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:45:58.078 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613036708.096437_52.1 target= 2021-02-11T09:46:03.079 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:46:03.080 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:46:03.080 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:46:03.122 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:46:03.122 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:46:03.346 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:46:03.346 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613036708.096437_52.1 target= 2021-02-11T09:46:08.347 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:46:08.347 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:46:08.348 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:46:08.390 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:46:08.390 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:46:08.613 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:46:08.614 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613036708.096437_52.1 target= 2021-02-11T09:46:13.614 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:46:13.615 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:46:13.615 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:46:13.657 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:46:13.657 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:46:13.903 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:46:13.903 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613036708.096437_52.1 target= 2021-02-11T09:46:18.903 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:46:18.904 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:46:18.904 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:46:18.944 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:46:18.944 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:46:19.203 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:46:19.203 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613036708.096437_52.1 target= 2021-02-11T09:46:24.204 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:46:24.204 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:46:24.205 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:46:24.246 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:46:24.246 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:46:24.479 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:46:24.479 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613036708.096437_52.1 target= 2021-02-11T09:46:29.480 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:46:29.480 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:46:29.481 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:46:29.520 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:46:29.520 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:46:29.753 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:46:29.753 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613036708.096437_52.1 target= 2021-02-11T09:46:34.754 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:46:34.754 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:46:34.755 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:46:34.796 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:46:34.796 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:46:35.030 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:46:51.003 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613036708.096437_52.1 target= 2021-02-11T09:46:56.004 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:46:56.004 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:46:56.005 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:46:56.044 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:46:56.044 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:46:56.278 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:46:56.278 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613036708.096437_52.1 target= 2021-02-11T09:47:01.279 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:01.279 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:01.279 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:47:01.321 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:47:01.321 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:47:01.551 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:47:01.552 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613036708.096437_52.1 target= 2021-02-11T09:47:06.552 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:17.590 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:17.590 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:47:17.632 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:47:17.632 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:47:17.866 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036708.096437_52.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:45:08.096437Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:47:17.866 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T09:47:17.866 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T09:47:17.866 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:17.866 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:17.867 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:47:17.907 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:47:17.907 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:47:18.119 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036708.096437_52.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:45:08.096437Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:47:18.120 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T09:47:18.120 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T09:47:18.120 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:18.120 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:18.120 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:47:18.161 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:47:18.161 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:47:18.389 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036708.096437_52.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:45:08.096437Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:47:18.390 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T09:47:18.390 INFO:rgw_multi.tests:meta checkpoint 2021-02-11T09:47:18.390 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:18.390 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:18.390 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:47:18.429 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:47:18.429 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:47:18.610 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1 2021-02-11T09:47:18.645 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613036708.096437_52.1","last_update":"2021-02-11T09:45:08.096437Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T09:47:18.646 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T09:47:18.646 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:18.646 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:18.646 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:47:18.687 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:47:18.687 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:47:18.933 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036708.096437_52.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:45:08.096437Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:47:18.933 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T09:47:18.933 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T09:47:18.933 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:18.933 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:18.934 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:47:18.974 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:47:18.974 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:47:19.180 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036708.096437_52.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:45:08.096437Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:47:19.180 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T09:47:19.181 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T09:47:19.181 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:19.181 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:19.181 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:47:19.220 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:47:19.220 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:47:19.435 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036708.096437_52.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:45:08.096437Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:47:19.436 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T09:47:19.436 INFO:tasks.rgw_multisite_tests:running rgw multisite tests on 'tasks.rgw_multi.tests' with args=[] 2021-02-11T09:47:19.444 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=tkscyf-1 2021-02-11T09:47:23.343 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=tkscyf-2 2021-02-11T09:47:27.381 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=tkscyf-3 2021-02-11T09:47:27.428 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:27.429 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:27.429 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:47:27.462 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:47:27.462 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:47:27.624 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1 2021-02-11T09:47:27.653 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613036708.096437_52.1","last_update":"2021-02-11T09:45:08.096437Z"},{"marker":"1_1613036847.414041_51.1","last_update":"2021-02-11T09:47:27.414041Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T09:47:27.654 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T09:47:27.654 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:27.654 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:27.654 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:47:27.689 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:47:27.689 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:47:27.922 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036708.096437_52.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:45:08.096437Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613036847.414041_51.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:27.414041Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:47:27.922 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T09:47:27.922 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T09:47:27.922 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:27.922 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:27.923 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:47:27.963 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:47:27.963 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:47:28.180 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036708.096437_52.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:45:08.096437Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613036847.414041_51.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:27.414041Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:47:28.180 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T09:47:28.180 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T09:47:28.180 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:28.180 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:28.180 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:47:28.221 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:47:28.222 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:47:28.457 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036708.096437_52.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:45:08.096437Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613036847.414041_51.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:27.414041Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:47:28.457 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T09:47:28.499 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_create ... ok 2021-02-11T09:47:28.499 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=tkscyf-4 2021-02-11T09:47:28.526 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=tkscyf-5 2021-02-11T09:47:28.575 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=tkscyf-6 2021-02-11T09:47:28.624 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:28.624 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:28.624 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:47:28.660 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:47:28.661 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:47:28.815 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1 2021-02-11T09:47:28.847 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613036708.096437_52.1","last_update":"2021-02-11T09:45:08.096437Z"},{"marker":"1_1613036848.613340_63.1","last_update":"2021-02-11T09:47:28.613340Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T09:47:28.848 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T09:47:28.848 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:28.848 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:28.848 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:47:28.883 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:47:28.883 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:47:29.119 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036708.096437_52.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:45:08.096437Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613036848.613340_63.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:28.613340Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:47:29.120 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T09:47:29.120 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T09:47:29.120 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:29.120 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:29.120 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:47:29.161 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:47:29.162 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:47:29.360 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036708.096437_52.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:45:08.096437Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613036848.613340_63.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:28.613340Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:47:29.360 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T09:47:29.360 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T09:47:29.361 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:29.361 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:29.361 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:47:29.399 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:47:29.400 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:47:29.629 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036708.096437_52.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:45:08.096437Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613036848.613340_63.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:28.613340Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:47:29.629 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T09:47:29.967 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:29.967 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:29.968 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:47:30.000 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:47:30.000 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:47:30.170 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1 2021-02-11T09:47:30.206 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613036708.096437_52.1","last_update":"2021-02-11T09:45:08.096437Z"},{"marker":"1_1613036848.613340_63.1","last_update":"2021-02-11T09:47:28.613340Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T09:47:30.206 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T09:47:30.206 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:30.206 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:30.207 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:47:30.245 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:47:30.245 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:47:30.457 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036708.096437_52.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:45:08.096437Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613036848.613340_63.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:28.613340Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:47:30.457 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T09:47:30.457 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T09:47:30.457 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:30.457 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:30.458 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:47:30.498 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:47:30.498 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:47:30.707 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036708.096437_52.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:45:08.096437Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613036848.613340_63.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:28.613340Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:47:30.707 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T09:47:30.707 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T09:47:30.707 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:30.707 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:30.708 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:47:30.748 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:47:30.748 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:47:30.995 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036708.096437_52.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:45:08.096437Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613036848.613340_63.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:28.613340Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:47:30.995 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T09:47:31.036 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_recreate ... ok 2021-02-11T09:47:31.036 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=tkscyf-7 2021-02-11T09:47:31.061 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=tkscyf-8 2021-02-11T09:47:31.111 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=tkscyf-9 2021-02-11T09:47:31.157 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:31.157 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:31.157 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:47:31.199 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:47:31.199 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:47:31.340 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1 2021-02-11T09:47:31.375 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613036708.096437_52.1","last_update":"2021-02-11T09:45:08.096437Z"},{"marker":"1_1613036851.144668_75.1","last_update":"2021-02-11T09:47:31.144668Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T09:47:31.376 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T09:47:31.376 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:31.376 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:31.376 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:47:31.415 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:47:31.415 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:47:31.649 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036708.096437_52.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:45:08.096437Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613036851.138269_73.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:31.138269Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:47:31.650 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1613036851.144668_75.1 target=1_1613036851.138269_73.1 2021-02-11T09:47:36.651 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:36.651 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:36.652 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:47:36.692 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:47:36.692 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:47:36.917 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036708.096437_52.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:45:08.096437Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613036851.138269_73.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:31.138269Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:47:36.917 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1613036851.144668_75.1 target=1_1613036851.138269_73.1 2021-02-11T09:47:41.918 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:43.030 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:43.031 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:47:43.071 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:47:43.071 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:47:43.296 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036708.096437_52.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:45:08.096437Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613036851.144668_75.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:31.144668Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:47:43.297 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T09:47:43.297 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T09:47:43.297 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:43.297 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:43.297 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:47:43.330 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:47:43.330 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:47:43.541 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036708.096437_52.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:45:08.096437Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613036851.144668_75.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:31.144668Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:47:43.541 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T09:47:43.541 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T09:47:43.541 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:43.541 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:43.541 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:47:43.580 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:47:43.580 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:47:43.797 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036708.096437_52.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:45:08.096437Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613036851.144668_75.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:31.144668Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:47:43.797 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T09:47:44.127 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:44.127 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:44.127 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:47:44.162 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:47:44.162 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:47:44.327 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1 2021-02-11T09:47:44.360 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613036708.096437_52.1","last_update":"2021-02-11T09:45:08.096437Z"},{"marker":"1_1613036864.081584_81.1","last_update":"2021-02-11T09:47:44.081584Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T09:47:44.361 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T09:47:44.361 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:44.361 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:44.361 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:47:44.400 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:47:44.400 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:47:44.622 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036708.096437_52.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:45:08.096437Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:47:44.623 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T09:47:44.623 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T09:47:44.623 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:44.623 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:44.623 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:47:44.666 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:47:44.666 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:47:44.873 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036708.096437_52.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:45:08.096437Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:47:44.874 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T09:47:44.874 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T09:47:44.874 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:44.874 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:44.874 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:47:44.914 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:47:44.914 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:47:45.148 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036708.096437_52.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:45:08.096437Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:47:45.148 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T09:47:45.164 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_remove ... ok 2021-02-11T09:47:45.164 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=tkscyf-10 2021-02-11T09:47:45.195 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=tkscyf-11 2021-02-11T09:47:45.247 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=tkscyf-12 2021-02-11T09:47:45.385 INFO:tasks.rgw.c2.client.0.smithi098.stdout:2021-02-11T09:47:45.386+0000 7f2bdaffd700 -1 data sync zone:dd308248 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T09:47:45.385 INFO:tasks.rgw.c2.client.0.smithi098.stdout:2021-02-11T09:47:45.386+0000 7f2bb67ec700 -1 data sync zone:dd308248 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T09:47:45.385 INFO:tasks.rgw.c2.client.1.smithi098.stdout:2021-02-11T09:47:45.386+0000 7fa80efed700 -1 data sync zone:dd308248 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T09:47:45.386 INFO:tasks.rgw.c2.client.1.smithi098.stdout:2021-02-11T09:47:45.386+0000 7fa8167fc700 -1 data sync zone:dd308248 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T09:47:45.386 INFO:tasks.rgw.c1.client.1.smithi094.stdout:2021-02-11T09:47:45.389+0000 7fd3c87f8700 -1 data sync zone:dd308248 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T09:47:45.386 INFO:tasks.rgw.c1.client.1.smithi094.stdout:2021-02-11T09:47:45.389+0000 7fd3ca7fc700 -1 data sync zone:dd308248 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T09:47:45.874 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:45.875 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:45.875 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:47:45.912 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:47:45.912 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:47:46.076 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1 2021-02-11T09:47:46.101 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613036865.278468_73.1","last_update":"2021-02-11T09:47:45.278468Z"},{"marker":"1_1613036864.081584_81.1","last_update":"2021-02-11T09:47:44.081584Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T09:47:46.102 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T09:47:46.102 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:46.102 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:46.102 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:47:46.138 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:47:46.138 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:47:46.378 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036865.278468_73.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:47:45.278468Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:47:46.379 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T09:47:46.379 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T09:47:46.379 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:46.379 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:46.379 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:47:46.422 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:47:46.422 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:47:46.591 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036865.278468_73.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:47:45.278468Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:47:46.591 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T09:47:46.591 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T09:47:46.591 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:46.591 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:47:46.591 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:47:46.631 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:47:46.631 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:47:46.864 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036865.278468_73.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:47:45.278468Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:47:46.865 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T09:47:46.865 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:47:46.865 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:47:46.865 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:47:46.950 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:47:46.950 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:47:47.166 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:47:47.170+0000 7f485faad080 1 bucket sync caught up with source: 2021-02-11T09:47:47.167 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, 00000000001.5.5, , 00000000001.5.5, , , , 00000000001.6.5, 00000000001.6.5, , ] 2021-02-11T09:47:47.167 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, 00000000001.5.5, , 00000000001.5.5, , , , 00000000001.6.5, 00000000001.6.5, , ] 2021-02-11T09:47:47.167 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:47:47.170+0000 7f485faad080 0 bucket checkpoint complete 2021-02-11T09:47:47.196 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-10 zones={test-zone2, test-zone1} 2021-02-11T09:47:47.207 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:47:47.216 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-02-11T09:47:47.216 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-02-11T09:47:47.216 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-02-11T09:47:47.216 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T09:47:47.216 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:47:47.263 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-02-11T09:47:47.263 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-02-11T09:47:47.263 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-02-11T09:47:47.263 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T09:47:47.320 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-11T09:47:47.376 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-11T09:47:47.487 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-11T09:47:47.589 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-11T09:47:47.644 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-11T09:47:47.756 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-11T09:47:47.865 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-11T09:47:47.919 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-11T09:47:48.030 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-11T09:47:48.133 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T09:47:48.188 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T09:47:48.300 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T09:47:48.404 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-10 zones={test-zone2, test-zone1} 2021-02-11T09:47:48.405 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:47:48.405 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:47:48.405 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:47:48.449 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:47:48.449 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:47:48.644 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:47:48.649+0000 7fdfd4586080 1 bucket sync caught up with source: 2021-02-11T09:47:48.644 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000001.5.5, , 00000000001.5.5, , , , 00000000001.6.5, 00000000001.6.5, , ] 2021-02-11T09:47:48.644 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000001.5.5, , 00000000001.5.5, , , , 00000000001.6.5, 00000000001.6.5, , ] 2021-02-11T09:47:48.644 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:47:48.649+0000 7fdfd4586080 0 bucket checkpoint complete 2021-02-11T09:47:48.671 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-10 zones={test-zone3, test-zone1} 2021-02-11T09:47:48.680 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:47:48.685 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-02-11T09:47:48.685 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-02-11T09:47:48.686 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-02-11T09:47:48.686 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T09:47:48.686 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:47:48.690 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-02-11T09:47:48.690 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-02-11T09:47:48.690 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-02-11T09:47:48.690 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T09:47:48.700 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-11T09:47:48.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-11T09:47:48.858 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-11T09:47:48.961 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-11T09:47:49.015 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-11T09:47:49.129 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-11T09:47:49.233 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-11T09:47:49.285 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-11T09:47:49.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-11T09:47:49.498 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T09:47:49.551 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T09:47:49.658 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T09:47:49.764 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-10 zones={test-zone3, test-zone1} 2021-02-11T09:47:49.764 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:47:49.765 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:47:49.765 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:47:49.811 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:47:49.811 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:47:50.018 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:47:50.022+0000 7f932735a080 1 bucket sync caught up with source: 2021-02-11T09:47:50.019 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, 00000000001.5.5, , 00000000001.5.5, , , , 00000000001.6.5, 00000000001.6.5, , ] 2021-02-11T09:47:50.019 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, 00000000001.5.5, , 00000000001.5.5, , , , 00000000001.6.5, 00000000001.6.5, , ] 2021-02-11T09:47:50.019 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:47:50.022+0000 7f932735a080 0 bucket checkpoint complete 2021-02-11T09:47:50.055 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:47:50.055 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:47:50.055 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-11 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:47:50.098 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:47:50.098 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:47:50.287 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:47:50.289+0000 7f8a814b6080 1 bucket sync caught up with source: 2021-02-11T09:47:50.287 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000001.7.5, , 00000000001.9.5, , , , 00000000001.7.5, 00000000001.9.5, , ] 2021-02-11T09:47:50.287 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000001.7.5, , 00000000001.9.5, , , , 00000000001.7.5, 00000000001.9.5, , ] 2021-02-11T09:47:50.287 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:47:50.289+0000 7f8a814b6080 0 bucket checkpoint complete 2021-02-11T09:47:50.318 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-11 zones={test-zone1, test-zone2} 2021-02-11T09:47:50.329 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:47:50.334 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-02-11T09:47:50.335 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-02-11T09:47:50.335 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-02-11T09:47:50.335 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T09:47:50.335 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:47:50.342 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-02-11T09:47:50.342 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-02-11T09:47:50.342 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-02-11T09:47:50.342 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T09:47:50.351 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-11T09:47:50.407 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-11T09:47:50.515 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-11T09:47:50.617 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-11T09:47:50.671 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-11T09:47:50.821 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-11T09:47:50.925 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-11T09:47:50.979 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-11T09:47:51.091 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-11T09:47:51.197 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T09:47:51.252 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T09:47:51.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T09:47:51.507 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-11 zones={test-zone1, test-zone2} 2021-02-11T09:47:51.507 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:47:51.507 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:47:51.507 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-11 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:47:51.555 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:47:51.555 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:47:51.755 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:47:51.757+0000 7f5de8187080 1 bucket sync caught up with source: 2021-02-11T09:47:51.755 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000001.7.5, , 00000000001.9.5, , , , 00000000001.7.5, 00000000001.9.5, , ] 2021-02-11T09:47:51.755 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000001.7.5, , 00000000001.9.5, , , , 00000000001.7.5, 00000000001.9.5, , ] 2021-02-11T09:47:51.755 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:47:51.757+0000 7f5de8187080 0 bucket checkpoint complete 2021-02-11T09:47:51.785 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-11 zones={test-zone3, test-zone2} 2021-02-11T09:47:51.795 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:47:51.800 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-02-11T09:47:51.800 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-02-11T09:47:51.800 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-02-11T09:47:51.800 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T09:47:51.800 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:47:51.847 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-02-11T09:47:51.847 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-02-11T09:47:51.847 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-02-11T09:47:51.847 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T09:47:51.903 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-11T09:47:51.956 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-11T09:47:52.067 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-11T09:47:52.174 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-11T09:47:52.226 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-11T09:47:52.335 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-11T09:47:52.442 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-11T09:47:52.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-11T09:47:52.607 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-11T09:47:52.750 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T09:47:52.804 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T09:47:52.915 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T09:47:53.017 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-11 zones={test-zone3, test-zone2} 2021-02-11T09:47:53.017 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:47:53.017 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:47:53.017 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-12 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:47:53.067 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:47:53.067 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:47:53.269 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:47:53.273+0000 7fc9c3ce3080 1 bucket sync caught up with source: 2021-02-11T09:47:53.269 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000001.14.5, , 00000000001.5.5, , , , 00000000001.5.5, 00000000001.7.5, , ] 2021-02-11T09:47:53.269 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000001.14.5, , 00000000001.5.5, , , , 00000000001.5.5, 00000000001.7.5, , ] 2021-02-11T09:47:53.269 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:47:53.273+0000 7fc9c3ce3080 0 bucket checkpoint complete 2021-02-11T09:47:53.299 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-12 zones={test-zone1, test-zone3} 2021-02-11T09:47:53.309 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:47:53.355 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-02-11T09:47:53.355 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-02-11T09:47:53.355 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-02-11T09:47:53.356 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T09:47:53.356 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:47:53.362 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-02-11T09:47:53.362 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-02-11T09:47:53.362 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-02-11T09:47:53.362 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T09:47:53.372 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-11T09:47:53.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-11T09:47:53.530 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-11T09:47:53.633 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-11T09:47:53.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-11T09:47:53.794 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-11T09:47:53.897 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-11T09:47:53.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-11T09:47:54.058 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-11T09:47:54.160 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T09:47:54.217 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T09:47:54.327 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T09:47:54.429 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-12 zones={test-zone1, test-zone3} 2021-02-11T09:47:54.429 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:47:54.429 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:47:54.429 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-12 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:47:54.476 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:47:54.476 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:47:54.691 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:47:54.694+0000 7f4ead11d080 1 waiting for incremental sync to catch up: 2021-02-11T09:47:54.691 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, , , 00000000001.5.5, , , , 00000000001.5.5, 00000000001.7.5, , ] 2021-02-11T09:47:54.692 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, 00000000001.14.5, , 00000000001.5.5, , , , 00000000001.5.5, 00000000001.7.5, , ] 2021-02-11T09:47:59.695 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:47:59.698+0000 7f4ead11d080 1 waiting for incremental sync to catch up: 2021-02-11T09:47:59.696 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, , , 00000000001.5.5, , , , 00000000001.5.5, 00000000001.7.5, , ] 2021-02-11T09:47:59.696 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, 00000000001.14.5, , 00000000001.5.5, , , , 00000000001.5.5, 00000000001.7.5, , ] 2021-02-11T09:48:04.699 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:48:04.702+0000 7f4ead11d080 1 bucket sync caught up with source: 2021-02-11T09:48:04.699 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, 00000000001.14.5, , 00000000001.5.5, , , , 00000000001.5.5, 00000000001.7.5, , ] 2021-02-11T09:48:04.699 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, 00000000001.14.5, , 00000000001.5.5, , , , 00000000001.5.5, 00000000001.7.5, , ] 2021-02-11T09:48:04.699 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:48:04.702+0000 7f4ead11d080 0 bucket checkpoint complete 2021-02-11T09:48:04.733 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-12 zones={test-zone2, test-zone3} 2021-02-11T09:48:04.743 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:48:04.749 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-02-11T09:48:04.749 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-02-11T09:48:04.749 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-02-11T09:48:04.749 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T09:48:04.749 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:48:04.756 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-02-11T09:48:04.756 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-02-11T09:48:04.756 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-02-11T09:48:04.756 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T09:48:04.767 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-11T09:48:04.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-11T09:48:04.928 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-11T09:48:05.034 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-11T09:48:05.087 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-11T09:48:05.196 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-11T09:48:05.299 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-11T09:48:05.353 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-11T09:48:05.459 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-11T09:48:05.566 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T09:48:05.619 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T09:48:05.728 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T09:48:05.837 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-12 zones={test-zone2, test-zone3} 2021-02-11T09:48:05.838 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_object_sync ... ok 2021-02-11T09:48:05.838 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=tkscyf-13 2021-02-11T09:48:05.866 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=tkscyf-14 2021-02-11T09:48:05.917 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=tkscyf-15 2021-02-11T09:48:05.990 INFO:tasks.rgw.c1.client.1.smithi094.stdout:2021-02-11T09:48:05.993+0000 7fd3e9d63700 -1 data sync zone:dd308248 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T09:48:05.990 INFO:tasks.rgw.c2.client.1.smithi098.stdout:2021-02-11T09:48:05.994+0000 7fa80cfe9700 -1 data sync zone:dd308248 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T09:48:05.991 INFO:tasks.rgw.c2.client.0.smithi098.stdout:2021-02-11T09:48:05.994+0000 7f2bb37e6700 -1 data sync zone:dd308248 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T09:48:06.114 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:48:06.115 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:48:06.115 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:48:06.154 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:48:06.154 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:48:06.307 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1 2021-02-11T09:48:06.346 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613036885.953989_189.1","last_update":"2021-02-11T09:48:05.953989Z"},{"marker":"1_1613036864.081584_81.1","last_update":"2021-02-11T09:47:44.081584Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T09:48:06.347 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T09:48:06.347 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:48:06.347 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:48:06.347 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:48:06.382 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:48:06.382 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:48:06.595 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036885.953989_189.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:48:05.953989Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:48:06.596 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T09:48:06.596 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T09:48:06.596 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:48:06.596 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:48:06.596 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:48:06.638 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:48:06.638 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:48:06.839 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036885.953989_189.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:48:05.953989Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:48:06.839 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T09:48:06.839 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T09:48:06.840 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:48:06.840 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:48:06.840 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:48:06.881 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:48:06.881 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:48:07.109 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036885.953989_189.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:48:05.953989Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:48:07.110 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T09:48:07.110 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:07.111 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:07.111 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:48:07.209 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:48:07.209 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:48:07.394 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:48:07.398+0000 7f0a8bb37080 1 bucket sync caught up with source: 2021-02-11T09:48:07.394 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, 00000000001.6.5, , , , , , , , , ] 2021-02-11T09:48:07.394 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, 00000000001.6.5, , , , , , , , , ] 2021-02-11T09:48:07.394 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:48:07.398+0000 7f0a8bb37080 0 bucket checkpoint complete 2021-02-11T09:48:07.421 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-13 zones={test-zone2, test-zone1} 2021-02-11T09:48:07.432 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:48:07.479 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T09:48:07.479 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:48:07.487 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T09:48:07.545 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T09:48:07.599 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T09:48:07.707 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T09:48:07.813 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-13 zones={test-zone2, test-zone1} 2021-02-11T09:48:07.814 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:07.814 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:07.814 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:48:07.857 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:48:07.857 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:48:08.052 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:48:08.057+0000 7f1450a35080 1 bucket sync caught up with source: 2021-02-11T09:48:08.052 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000001.6.5, , , , , , , , , ] 2021-02-11T09:48:08.052 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000001.6.5, , , , , , , , , ] 2021-02-11T09:48:08.052 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:48:08.057+0000 7f1450a35080 0 bucket checkpoint complete 2021-02-11T09:48:08.078 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-13 zones={test-zone3, test-zone1} 2021-02-11T09:48:08.088 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:48:08.135 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T09:48:08.135 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:48:08.141 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T09:48:08.196 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T09:48:08.252 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T09:48:08.361 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T09:48:08.466 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-13 zones={test-zone3, test-zone1} 2021-02-11T09:48:08.466 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:08.466 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:08.466 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:48:08.514 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:48:08.514 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:48:08.716 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:48:08.718+0000 7f10343a5080 1 bucket sync caught up with source: 2021-02-11T09:48:08.716 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, 00000000001.6.5, , , , , , , , , ] 2021-02-11T09:48:08.716 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, 00000000001.6.5, , , , , , , , , ] 2021-02-11T09:48:08.716 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:48:08.718+0000 7f10343a5080 0 bucket checkpoint complete 2021-02-11T09:48:08.747 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:08.747 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:08.747 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:48:08.796 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:48:08.797 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:48:08.992 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:48:08.997+0000 7f6ac0dca080 1 bucket sync caught up with source: 2021-02-11T09:48:08.992 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000001.12.5, , , , , , , , , ] 2021-02-11T09:48:08.992 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000001.12.5, , , , , , , , , ] 2021-02-11T09:48:08.992 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:48:08.997+0000 7f6ac0dca080 0 bucket checkpoint complete 2021-02-11T09:48:09.023 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-14 zones={test-zone1, test-zone2} 2021-02-11T09:48:09.034 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:48:09.083 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T09:48:09.083 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:48:09.089 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T09:48:09.145 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T09:48:09.199 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T09:48:09.308 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T09:48:09.414 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-14 zones={test-zone1, test-zone2} 2021-02-11T09:48:09.414 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:09.414 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:09.414 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:48:09.460 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:48:09.460 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:48:09.660 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:48:09.665+0000 7f1d0dea5080 1 bucket sync caught up with source: 2021-02-11T09:48:09.660 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000001.12.5, , , , , , , , , ] 2021-02-11T09:48:09.660 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000001.12.5, , , , , , , , , ] 2021-02-11T09:48:09.660 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:48:09.665+0000 7f1d0dea5080 0 bucket checkpoint complete 2021-02-11T09:48:09.692 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-14 zones={test-zone3, test-zone2} 2021-02-11T09:48:09.702 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:48:09.751 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T09:48:09.751 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:48:09.758 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T09:48:09.812 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T09:48:09.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T09:48:09.979 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T09:48:10.081 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-14 zones={test-zone3, test-zone2} 2021-02-11T09:48:10.081 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:10.081 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:10.082 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:48:10.130 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:48:10.130 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:48:10.332 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:48:10.337+0000 7f45e2107080 1 bucket sync caught up with source: 2021-02-11T09:48:10.332 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000001.18.5, , , , , , , , , ] 2021-02-11T09:48:10.332 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000001.18.5, , , , , , , , , ] 2021-02-11T09:48:10.332 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:48:10.337+0000 7f45e2107080 0 bucket checkpoint complete 2021-02-11T09:48:10.358 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-15 zones={test-zone1, test-zone3} 2021-02-11T09:48:10.367 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:48:10.415 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T09:48:10.415 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:48:10.421 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T09:48:10.477 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T09:48:10.530 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T09:48:10.639 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T09:48:10.737 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-15 zones={test-zone1, test-zone3} 2021-02-11T09:48:10.737 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:10.737 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:10.737 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:48:10.786 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:48:10.787 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:48:11.009 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:48:11.014+0000 7fa118aa3080 1 bucket sync caught up with source: 2021-02-11T09:48:11.009 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, 00000000001.18.5, , , , , , , , , ] 2021-02-11T09:48:11.010 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, 00000000001.18.5, , , , , , , , , ] 2021-02-11T09:48:11.010 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:48:11.014+0000 7fa118aa3080 0 bucket checkpoint complete 2021-02-11T09:48:11.041 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-15 zones={test-zone2, test-zone3} 2021-02-11T09:48:11.051 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:48:11.099 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T09:48:11.099 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:48:11.105 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T09:48:11.161 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T09:48:11.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T09:48:11.326 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T09:48:11.431 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-15 zones={test-zone2, test-zone3} 2021-02-11T09:48:11.445 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:11.445 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:11.445 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:48:11.495 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:48:11.495 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:48:11.704 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:48:11.706+0000 7fdfc3ae5080 1 bucket sync caught up with source: 2021-02-11T09:48:11.705 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, 00000000002.8.5, , , , , , , , , ] 2021-02-11T09:48:11.705 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, 00000000002.8.5, , , , , , , , , ] 2021-02-11T09:48:11.705 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:48:11.706+0000 7fdfc3ae5080 0 bucket checkpoint complete 2021-02-11T09:48:11.739 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-13 zones={test-zone2, test-zone1} 2021-02-11T09:48:11.749 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:48:11.794 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:48:11.860 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-13 zones={test-zone2, test-zone1} 2021-02-11T09:48:11.861 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:11.861 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:11.861 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:48:11.904 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:48:11.904 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:48:12.098 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:48:12.101+0000 7fdf59db7080 1 bucket sync caught up with source: 2021-02-11T09:48:12.098 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000002.8.5, , , , , , , , , ] 2021-02-11T09:48:12.098 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000002.8.5, , , , , , , , , ] 2021-02-11T09:48:12.098 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:48:12.101+0000 7fdf59db7080 0 bucket checkpoint complete 2021-02-11T09:48:12.126 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-13 zones={test-zone3, test-zone1} 2021-02-11T09:48:12.840 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:48:12.886 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:48:12.944 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-13 zones={test-zone3, test-zone1} 2021-02-11T09:48:12.944 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:12.944 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:12.944 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:48:12.992 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:48:12.992 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:48:13.190 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:48:13.194+0000 7f0f0a32f080 1 bucket sync caught up with source: 2021-02-11T09:48:13.190 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, 00000000002.8.5, , , , , , , , , ] 2021-02-11T09:48:13.190 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, 00000000002.8.5, , , , , , , , , ] 2021-02-11T09:48:13.190 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:48:13.194+0000 7f0f0a32f080 0 bucket checkpoint complete 2021-02-11T09:48:13.243 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:13.243 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:13.243 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:48:13.291 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:48:13.291 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:48:13.483 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:48:13.485+0000 7f11aa949080 1 bucket sync caught up with source: 2021-02-11T09:48:13.483 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000002.14.5, , , , , , , , , ] 2021-02-11T09:48:13.483 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000002.14.5, , , , , , , , , ] 2021-02-11T09:48:13.483 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:48:13.485+0000 7f11aa949080 0 bucket checkpoint complete 2021-02-11T09:48:13.505 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-14 zones={test-zone1, test-zone2} 2021-02-11T09:48:13.519 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:48:13.566 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:48:13.629 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-14 zones={test-zone1, test-zone2} 2021-02-11T09:48:13.629 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:13.629 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:13.630 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:48:13.675 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:48:13.676 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:48:13.868 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:48:13.873+0000 7f4eb831d080 1 bucket sync caught up with source: 2021-02-11T09:48:13.869 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000002.14.5, , , , , , , , , ] 2021-02-11T09:48:13.869 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000002.14.5, , , , , , , , , ] 2021-02-11T09:48:13.869 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:48:13.873+0000 7f4eb831d080 0 bucket checkpoint complete 2021-02-11T09:48:13.901 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-14 zones={test-zone3, test-zone2} 2021-02-11T09:48:13.912 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:48:13.958 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:48:14.022 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-14 zones={test-zone3, test-zone2} 2021-02-11T09:48:14.039 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:14.039 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:14.039 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:48:14.080 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:48:14.081 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:48:14.267 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:48:14.269+0000 7f0841de6080 1 waiting for incremental sync to catch up: 2021-02-11T09:48:14.267 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000001.18.5, , , , , , , , , ] 2021-02-11T09:48:14.267 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000002.20.5, , , , , , , , , ] 2021-02-11T09:48:19.269 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:48:19.273+0000 7f0841de6080 1 bucket sync caught up with source: 2021-02-11T09:48:19.270 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000002.20.5, , , , , , , , , ] 2021-02-11T09:48:19.270 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000002.20.5, , , , , , , , , ] 2021-02-11T09:48:19.270 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:48:19.273+0000 7f0841de6080 0 bucket checkpoint complete 2021-02-11T09:48:19.301 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-15 zones={test-zone1, test-zone3} 2021-02-11T09:48:19.308 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:48:19.355 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:48:19.416 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-15 zones={test-zone1, test-zone3} 2021-02-11T09:48:19.416 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:19.416 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:19.416 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:48:19.463 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:48:19.463 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:48:19.683 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:48:19.686+0000 7f13b20a7080 1 bucket sync caught up with source: 2021-02-11T09:48:19.683 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, 00000000002.20.5, , , , , , , , , ] 2021-02-11T09:48:19.683 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, 00000000002.20.5, , , , , , , , , ] 2021-02-11T09:48:19.684 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:48:19.686+0000 7f13b20a7080 0 bucket checkpoint complete 2021-02-11T09:48:19.714 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-15 zones={test-zone2, test-zone3} 2021-02-11T09:48:19.724 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:48:19.770 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:48:19.833 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-15 zones={test-zone2, test-zone3} 2021-02-11T09:48:19.833 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_object_delete ... ok 2021-02-11T09:48:19.833 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=tkscyf-16 2021-02-11T09:48:19.863 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=tkscyf-17 2021-02-11T09:48:19.918 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=tkscyf-18 2021-02-11T09:48:20.031 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:48:20.031 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:48:20.031 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:48:20.067 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:48:20.067 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:48:20.229 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1 2021-02-11T09:48:20.268 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613036900.015316_211.1","last_update":"2021-02-11T09:48:20.015316Z"},{"marker":"1_1613036864.081584_81.1","last_update":"2021-02-11T09:47:44.081584Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T09:48:20.268 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T09:48:20.269 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:48:20.269 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:48:20.269 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:48:20.303 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:48:20.303 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:48:20.504 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036900.015316_211.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:48:20.015316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:48:20.504 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T09:48:20.505 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T09:48:20.505 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:48:20.505 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:48:20.505 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:48:20.544 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:48:20.544 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:48:20.762 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036900.015316_211.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:48:20.015316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:48:20.762 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T09:48:20.762 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T09:48:20.762 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:48:20.762 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:48:20.762 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:48:20.804 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:48:20.804 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:48:21.034 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036900.015316_211.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:48:20.015316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:48:21.035 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T09:48:21.086 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:21.087 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:21.087 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:48:21.135 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:48:21.135 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:48:21.336 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:48:21.338+0000 7fccaf0ad080 1 waiting for incremental sync to catch up: 2021-02-11T09:48:21.336 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, , , , , , , , , , ] 2021-02-11T09:48:21.336 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, , , , , 00000000002.8.10, , , , , ] 2021-02-11T09:48:26.340 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:48:26.342+0000 7fccaf0ad080 1 bucket sync caught up with source: 2021-02-11T09:48:26.340 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, , , , , 00000000006.16.9, , , , , ] 2021-02-11T09:48:26.340 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, , , , , 00000000002.8.10, , , , , ] 2021-02-11T09:48:26.340 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:48:26.342+0000 7fccaf0ad080 0 bucket checkpoint complete 2021-02-11T09:48:26.374 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:26.374 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:26.374 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:48:26.424 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:48:26.424 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:48:26.614 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:48:26.617+0000 7fe33ed94080 1 bucket sync caught up with source: 2021-02-11T09:48:26.614 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , 00000000006.16.9, , , , , ] 2021-02-11T09:48:26.615 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , 00000000006.16.9, , , , , ] 2021-02-11T09:48:26.615 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:48:26.617+0000 7fe33ed94080 0 bucket checkpoint complete 2021-02-11T09:48:26.643 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:26.643 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:26.643 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:48:26.690 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:48:26.690 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:48:26.899 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:48:26.902+0000 7feba1019080 1 bucket sync caught up with source: 2021-02-11T09:48:26.899 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, , , , , 00000000006.16.9, , , , , ] 2021-02-11T09:48:26.899 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, , , , , 00000000006.16.9, , , , , ] 2021-02-11T09:48:26.899 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:48:26.902+0000 7feba1019080 0 bucket checkpoint complete 2021-02-11T09:48:26.978 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:26.979 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:26.979 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:48:27.027 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:48:27.027 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:48:27.218 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:48:27.221+0000 7efd45b8d080 1 bucket sync caught up with source: 2021-02-11T09:48:27.218 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , 00000000002.13.10, , , , , ] 2021-02-11T09:48:27.218 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , 00000000002.13.10, , , , , ] 2021-02-11T09:48:27.218 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:48:27.221+0000 7efd45b8d080 0 bucket checkpoint complete 2021-02-11T09:48:27.243 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:27.243 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:27.243 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:48:27.286 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:48:27.286 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:48:27.444 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:48:27.449+0000 7f0d7947a080 1 waiting for incremental sync to catch up: 2021-02-11T09:48:27.444 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , 00000000002.13.10, , , , , ] 2021-02-11T09:48:27.444 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , 00000000006.21.9, , , , , ] 2021-02-11T09:48:32.448 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:48:32.453+0000 7f0d7947a080 1 bucket sync caught up with source: 2021-02-11T09:48:32.448 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , 00000000006.21.9, , , , , ] 2021-02-11T09:48:32.448 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , 00000000006.21.9, , , , , ] 2021-02-11T09:48:32.448 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:48:32.453+0000 7f0d7947a080 0 bucket checkpoint complete 2021-02-11T09:48:32.523 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:32.523 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:32.523 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:48:32.569 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:48:32.569 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:48:32.747 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:48:32.753+0000 7fefeda99080 1 waiting for incremental sync to catch up: 2021-02-11T09:48:32.747 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , , , ] 2021-02-11T09:48:32.747 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , 00000000002.11.10, , , , , ] 2021-02-11T09:48:37.751 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:48:37.757+0000 7fefeda99080 1 bucket sync caught up with source: 2021-02-11T09:48:37.752 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , 00000000006.19.9, , , , , ] 2021-02-11T09:48:37.752 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , 00000000002.11.10, , , , , ] 2021-02-11T09:48:37.752 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:48:37.757+0000 7fefeda99080 0 bucket checkpoint complete 2021-02-11T09:48:37.786 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:37.787 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:37.789 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:48:37.846 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:48:37.846 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:48:38.059 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:48:38.062+0000 7ff3aa6fe080 1 bucket sync caught up with source: 2021-02-11T09:48:38.059 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, , , , , 00000000006.19.9, , , , , ] 2021-02-11T09:48:38.059 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, , , , , 00000000006.19.9, , , , , ] 2021-02-11T09:48:38.059 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:48:38.062+0000 7ff3aa6fe080 0 bucket checkpoint complete 2021-02-11T09:48:39.722 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:39.723 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:39.723 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:48:39.770 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:48:39.770 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:48:39.968 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:48:39.970+0000 7f8da5632080 1 bucket sync caught up with source: 2021-02-11T09:48:39.968 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, 00000000010.25.5, , 00000000003.8.9, , 00000000009.22.9, , , , , ] 2021-02-11T09:48:39.968 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, 00000000010.25.5, , 00000000003.8.9, , 00000000009.22.9, , , , , ] 2021-02-11T09:48:39.968 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:48:39.970+0000 7f8da5632080 0 bucket checkpoint complete 2021-02-11T09:48:40.005 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:40.005 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:40.006 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:48:40.053 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:48:40.053 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:48:40.228 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:48:40.233+0000 7f2efad15080 1 bucket sync caught up with source: 2021-02-11T09:48:40.228 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000010.25.5, , 00000000003.8.9, , 00000000009.22.9, , , , , ] 2021-02-11T09:48:40.228 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000010.25.5, , 00000000003.8.9, , 00000000009.22.9, , , , , ] 2021-02-11T09:48:40.228 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:48:40.233+0000 7f2efad15080 0 bucket checkpoint complete 2021-02-11T09:48:40.255 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:40.255 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:40.255 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:48:40.303 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:48:40.304 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:48:40.496 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:48:40.498+0000 7f5326ac5080 1 bucket sync caught up with source: 2021-02-11T09:48:40.496 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, 00000000010.25.5, , 00000000003.8.9, , 00000000009.22.9, , , , , ] 2021-02-11T09:48:40.496 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, 00000000010.25.5, , 00000000003.8.9, , 00000000009.22.9, , , , , ] 2021-02-11T09:48:40.496 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:48:40.498+0000 7f5326ac5080 0 bucket checkpoint complete 2021-02-11T09:48:40.529 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:40.529 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:40.529 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:48:40.575 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:48:40.575 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:48:40.780 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:48:40.785+0000 7f2d50ad0080 1 bucket sync caught up with source: 2021-02-11T09:48:40.780 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000003.14.9, , 00000000010.24.5, , 00000000007.23.9, , , , , ] 2021-02-11T09:48:40.780 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000003.14.9, , 00000000010.24.5, , 00000000007.23.9, , , , , ] 2021-02-11T09:48:40.780 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:48:40.785+0000 7f2d50ad0080 0 bucket checkpoint complete 2021-02-11T09:48:40.814 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:40.814 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:40.814 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:48:40.864 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:48:40.864 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:48:41.066 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:48:41.069+0000 7faff484f080 1 bucket sync caught up with source: 2021-02-11T09:48:41.066 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000003.14.9, , 00000000010.24.5, , 00000000007.23.9, , , , , ] 2021-02-11T09:48:41.066 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000003.14.9, , 00000000010.24.5, , 00000000007.23.9, , , , , ] 2021-02-11T09:48:41.066 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:48:41.069+0000 7faff484f080 0 bucket checkpoint complete 2021-02-11T09:48:41.097 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:41.098 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:41.098 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:48:41.141 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:48:41.141 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:48:41.327 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:48:41.333+0000 7f2a28b0c080 1 bucket sync caught up with source: 2021-02-11T09:48:41.327 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000003.16.9, , 00000000003.12.9, , 00000000014.36.5, , , , , ] 2021-02-11T09:48:41.327 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000003.16.9, , 00000000003.12.9, , 00000000014.36.5, , , , , ] 2021-02-11T09:48:41.327 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:48:41.333+0000 7f2a28b0c080 0 bucket checkpoint complete 2021-02-11T09:48:41.352 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:41.352 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:41.352 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:48:41.401 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:48:41.401 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:48:41.612 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:48:41.614+0000 7fc92539a080 1 bucket sync caught up with source: 2021-02-11T09:48:41.612 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, 00000000003.16.9, , 00000000003.12.9, , 00000000014.36.5, , , , , ] 2021-02-11T09:48:41.613 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, 00000000003.16.9, , 00000000003.12.9, , 00000000014.36.5, , , , , ] 2021-02-11T09:48:41.613 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:48:41.614+0000 7fc92539a080 0 bucket checkpoint complete 2021-02-11T09:48:41.639 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-16 zones={test-zone2, test-zone1} 2021-02-11T09:48:41.648 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:48:41.655 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:48:41.655 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T09:48:41.655 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T09:48:41.655 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T09:48:41.655 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:48:41.703 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:48:41.703 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T09:48:41.703 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T09:48:41.703 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T09:48:41.759 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:48:41.770 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:48:41.835 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:48:41.891 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:48:41.950 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:48:42.063 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:48:42.166 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:48:42.219 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:48:42.333 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:48:42.438 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:48:42.491 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:48:42.606 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:48:42.709 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-16 zones={test-zone2, test-zone1} 2021-02-11T09:48:42.709 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-16 zones={test-zone3, test-zone1} 2021-02-11T09:48:42.719 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:48:42.723 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:48:42.723 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T09:48:42.724 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T09:48:42.724 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T09:48:42.724 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:48:42.728 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:48:42.728 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T09:48:42.728 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T09:48:42.728 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T09:48:42.736 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:48:42.744 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:48:42.807 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:48:42.865 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:48:43.327 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:48:43.438 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:48:43.542 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:48:43.596 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:48:43.709 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:48:43.812 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:48:43.868 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:48:43.981 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:48:44.085 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-16 zones={test-zone3, test-zone1} 2021-02-11T09:48:44.085 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-16 zones={test-zone3, test-zone2} 2021-02-11T09:48:44.096 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:48:44.101 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:48:44.102 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T09:48:44.102 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T09:48:44.102 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T09:48:44.102 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:48:44.109 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:48:44.109 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T09:48:44.109 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T09:48:44.109 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T09:48:44.119 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:48:44.129 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:48:44.197 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:48:44.257 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:48:44.311 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:48:44.425 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:48:44.529 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:48:44.584 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:48:44.699 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:48:44.806 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:48:44.860 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:48:44.974 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:48:45.077 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-16 zones={test-zone3, test-zone2} 2021-02-11T09:48:45.077 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:45.078 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:45.078 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:48:45.122 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:48:45.122 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:48:45.324 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:48:45.326+0000 7fdd08754080 1 bucket sync caught up with source: 2021-02-11T09:48:45.324 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, 00000000010.31.5, , 00000000003.10.9, , 00000000011.48.5, , , , , ] 2021-02-11T09:48:45.324 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, 00000000010.31.5, , 00000000003.10.9, , 00000000011.48.5, , , , , ] 2021-02-11T09:48:45.324 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:48:45.326+0000 7fdd08754080 0 bucket checkpoint complete 2021-02-11T09:48:45.356 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:45.356 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:45.356 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:48:45.401 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:48:45.401 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:48:45.594 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:48:45.597+0000 7f393de06080 1 bucket sync caught up with source: 2021-02-11T09:48:45.594 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000010.31.5, , 00000000003.10.9, , 00000000011.48.5, , , , , ] 2021-02-11T09:48:45.594 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000010.31.5, , 00000000003.10.9, , 00000000011.48.5, , , , , ] 2021-02-11T09:48:45.594 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:48:45.597+0000 7f393de06080 0 bucket checkpoint complete 2021-02-11T09:48:45.620 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:45.620 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:45.621 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:48:45.669 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:48:45.669 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:48:45.889 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:48:45.894+0000 7f895b9d5080 1 bucket sync caught up with source: 2021-02-11T09:48:45.889 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, 00000000010.31.5, , 00000000003.10.9, , 00000000011.48.5, , , , , ] 2021-02-11T09:48:45.890 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, 00000000010.31.5, , 00000000003.10.9, , 00000000011.48.5, , , , , ] 2021-02-11T09:48:45.890 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:48:45.894+0000 7f895b9d5080 0 bucket checkpoint complete 2021-02-11T09:48:45.922 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:45.922 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:45.923 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:48:45.971 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:48:45.971 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:48:46.163 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:48:46.169+0000 7f55a732d080 1 bucket sync caught up with source: 2021-02-11T09:48:46.163 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000003.32.9, , 00000000010.29.5, , 00000000012.40.5, , , , , ] 2021-02-11T09:48:46.163 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000003.32.9, , 00000000010.29.5, , 00000000012.40.5, , , , , ] 2021-02-11T09:48:46.163 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:48:46.169+0000 7f55a732d080 0 bucket checkpoint complete 2021-02-11T09:48:46.194 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:46.194 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:46.194 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:48:46.239 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:48:46.240 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:48:46.428 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:48:46.433+0000 7fe8625ce080 1 bucket sync caught up with source: 2021-02-11T09:48:46.428 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000003.32.9, , 00000000010.29.5, , 00000000012.40.5, , , , , ] 2021-02-11T09:48:46.429 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000003.32.9, , 00000000010.29.5, , 00000000012.40.5, , , , , ] 2021-02-11T09:48:46.429 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:48:46.433+0000 7fe8625ce080 0 bucket checkpoint complete 2021-02-11T09:48:46.462 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:46.462 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:46.462 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:48:46.505 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:48:46.505 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:48:46.698 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:48:46.701+0000 7f3b1b7c3080 1 bucket sync caught up with source: 2021-02-11T09:48:46.699 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000003.37.9, , 00000000003.14.9, , 00000000014.42.5, , , , , ] 2021-02-11T09:48:46.699 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000003.37.9, , 00000000003.14.9, , 00000000014.42.5, , , , , ] 2021-02-11T09:48:46.699 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:48:46.705+0000 7f3b1b7c3080 0 bucket checkpoint complete 2021-02-11T09:48:46.725 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:46.725 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:46.726 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:48:46.773 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:48:46.773 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:48:47.001 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:48:47.006+0000 7f9277c64080 1 bucket sync caught up with source: 2021-02-11T09:48:47.001 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, 00000000003.37.9, , 00000000003.14.9, , 00000000014.42.5, , , , , ] 2021-02-11T09:48:47.001 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, 00000000003.37.9, , 00000000003.14.9, , 00000000014.42.5, , , , , ] 2021-02-11T09:48:47.001 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:48:47.006+0000 7f9277c64080 0 bucket checkpoint complete 2021-02-11T09:48:47.031 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-17 zones={test-zone2, test-zone1} 2021-02-11T09:48:47.041 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:48:47.095 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:48:47.095 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T09:48:47.095 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T09:48:47.095 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T09:48:47.095 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:48:47.102 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:48:47.102 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T09:48:47.102 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T09:48:47.102 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T09:48:47.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:48:47.173 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:48:47.241 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:48:47.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:48:47.350 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:48:47.462 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:48:47.565 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:48:47.620 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:48:47.735 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:48:47.841 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:48:47.895 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:48:48.015 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:48:48.117 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-17 zones={test-zone2, test-zone1} 2021-02-11T09:48:48.117 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-17 zones={test-zone3, test-zone1} 2021-02-11T09:48:48.126 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:48:48.131 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:48:48.131 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T09:48:48.131 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T09:48:48.131 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T09:48:48.131 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:48:48.135 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:48:48.135 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T09:48:48.135 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T09:48:48.135 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T09:48:48.143 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:48:48.152 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:48:48.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:48:48.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:48:48.327 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:48:48.442 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:48:48.545 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:48:48.599 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:48:48.708 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:48:48.808 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:48:48.864 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:48:48.977 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:48:49.081 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-17 zones={test-zone3, test-zone1} 2021-02-11T09:48:49.081 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-17 zones={test-zone3, test-zone2} 2021-02-11T09:48:49.091 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:48:49.096 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:48:49.096 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T09:48:49.096 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T09:48:49.096 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T09:48:49.096 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:48:49.102 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:48:49.102 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T09:48:49.102 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T09:48:49.102 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T09:48:49.112 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:48:49.122 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:48:49.188 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:48:49.248 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:48:49.303 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:48:49.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:48:49.513 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:48:49.571 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:48:49.685 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:48:49.792 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:48:49.848 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:48:49.963 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:48:50.065 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-17 zones={test-zone3, test-zone2} 2021-02-11T09:48:50.065 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:50.065 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:50.065 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:48:50.106 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:48:50.106 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:48:50.309 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:48:50.314+0000 7f6836f45080 1 bucket sync caught up with source: 2021-02-11T09:48:50.309 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, 00000000010.26.5, , 00000000003.11.9, , 00000000007.17.9, , , , , ] 2021-02-11T09:48:50.309 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, 00000000010.26.5, , 00000000003.11.9, , 00000000007.17.9, , , , , ] 2021-02-11T09:48:50.309 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:48:50.314+0000 7f6836f45080 0 bucket checkpoint complete 2021-02-11T09:48:50.339 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:50.339 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:50.340 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:48:50.382 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:48:50.382 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:48:50.571 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:48:50.577+0000 7f011ec37080 1 bucket sync caught up with source: 2021-02-11T09:48:50.571 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000010.26.5, , 00000000003.11.9, , 00000000007.17.9, , , , , ] 2021-02-11T09:48:50.571 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000010.26.5, , 00000000003.11.9, , 00000000007.17.9, , , , , ] 2021-02-11T09:48:50.571 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:48:50.577+0000 7f011ec37080 0 bucket checkpoint complete 2021-02-11T09:48:50.598 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:50.598 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:50.599 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:48:50.648 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:48:50.649 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:48:50.862 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:48:50.866+0000 7f319ce69080 1 bucket sync caught up with source: 2021-02-11T09:48:50.862 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, 00000000010.26.5, , 00000000003.11.9, , 00000000007.17.9, , , , , ] 2021-02-11T09:48:50.862 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, 00000000010.26.5, , 00000000003.11.9, , 00000000007.17.9, , , , , ] 2021-02-11T09:48:50.862 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:48:50.866+0000 7f319ce69080 0 bucket checkpoint complete 2021-02-11T09:48:50.893 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:50.893 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:50.894 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:48:50.940 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:48:50.941 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:48:51.146 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:48:51.149+0000 7f6e61728080 1 bucket sync caught up with source: 2021-02-11T09:48:51.146 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000003.15.9, , 00000000010.30.5, , 00000000008.22.9, , , , , ] 2021-02-11T09:48:51.146 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000003.15.9, , 00000000010.30.5, , 00000000008.22.9, , , , , ] 2021-02-11T09:48:51.147 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:48:51.149+0000 7f6e61728080 0 bucket checkpoint complete 2021-02-11T09:48:51.179 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:51.179 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:51.179 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:48:51.225 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:48:51.225 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:48:51.410 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:48:51.413+0000 7f0266641080 1 bucket sync caught up with source: 2021-02-11T09:48:51.410 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000003.15.9, , 00000000010.30.5, , 00000000008.22.9, , , , , ] 2021-02-11T09:48:51.410 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000003.15.9, , 00000000010.30.5, , 00000000008.22.9, , , , , ] 2021-02-11T09:48:51.410 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:48:51.413+0000 7f0266641080 0 bucket checkpoint complete 2021-02-11T09:48:51.442 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:51.442 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:51.442 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:48:51.486 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:48:51.487 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:48:51.683 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:48:51.689+0000 7f1c7702a080 1 bucket sync caught up with source: 2021-02-11T09:48:51.683 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000003.19.9, , 00000000003.15.9, , 00000000016.39.5, , , , , ] 2021-02-11T09:48:51.684 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000003.19.9, , 00000000003.15.9, , 00000000016.39.5, , , , , ] 2021-02-11T09:48:51.684 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:48:51.689+0000 7f1c7702a080 0 bucket checkpoint complete 2021-02-11T09:48:51.711 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:51.711 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:51.712 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:48:51.758 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:48:51.758 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:48:51.965 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:48:51.966+0000 7f3a8ba2b080 1 bucket sync caught up with source: 2021-02-11T09:48:51.965 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, 00000000003.19.9, , 00000000003.15.9, , 00000000016.39.5, , , , , ] 2021-02-11T09:48:51.965 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, 00000000003.19.9, , 00000000003.15.9, , 00000000016.39.5, , , , , ] 2021-02-11T09:48:51.965 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:48:51.966+0000 7f3a8ba2b080 0 bucket checkpoint complete 2021-02-11T09:48:52.001 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-18 zones={test-zone2, test-zone1} 2021-02-11T09:48:52.011 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:48:52.017 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:48:52.017 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T09:48:52.017 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T09:48:52.017 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T09:48:52.017 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:48:52.023 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:48:52.023 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T09:48:52.023 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T09:48:52.023 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T09:48:52.033 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:48:52.044 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:48:52.109 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:48:52.169 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:48:52.225 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:48:52.343 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:48:52.445 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:48:52.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:48:52.616 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:48:52.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:48:52.776 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:48:52.892 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:48:53.001 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-18 zones={test-zone2, test-zone1} 2021-02-11T09:48:53.002 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-18 zones={test-zone3, test-zone1} 2021-02-11T09:48:53.009 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:48:53.013 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:48:53.014 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T09:48:53.014 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T09:48:53.014 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T09:48:53.014 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:48:53.018 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:48:53.018 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T09:48:53.018 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T09:48:53.018 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T09:48:53.027 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:48:53.035 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:48:53.103 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:48:53.160 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:48:53.216 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:48:53.330 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:48:53.433 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:48:53.487 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:48:53.601 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:48:53.704 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:48:53.759 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:48:53.873 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:48:53.981 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-18 zones={test-zone3, test-zone1} 2021-02-11T09:48:53.981 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-18 zones={test-zone3, test-zone2} 2021-02-11T09:48:53.992 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:48:53.997 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:48:53.997 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T09:48:53.997 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T09:48:53.997 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T09:48:53.997 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:48:54.003 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:48:54.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T09:48:54.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T09:48:54.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T09:48:54.014 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:48:54.023 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:48:54.088 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:48:54.149 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:48:54.203 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:48:54.313 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:48:54.416 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:48:54.476 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:48:54.592 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:48:54.697 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:48:54.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:48:54.860 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:48:54.965 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-18 zones={test-zone3, test-zone2} 2021-02-11T09:48:55.560 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:55.561 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:55.561 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:48:55.606 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:48:55.606 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:48:55.803 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:48:55.806+0000 7f3492c36080 1 bucket sync caught up with source: 2021-02-11T09:48:55.803 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, 00000000011.32.5, , 00000000004.11.5, , 00000000010.25.5, , , , , ] 2021-02-11T09:48:55.803 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, 00000000011.32.5, , 00000000004.11.5, , 00000000010.25.5, , , , , ] 2021-02-11T09:48:55.803 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:48:55.806+0000 7f3492c36080 0 bucket checkpoint complete 2021-02-11T09:48:55.830 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:55.830 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:55.831 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:48:55.882 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:48:55.883 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:48:56.073 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:48:56.077+0000 7fa572d7c080 1 bucket sync caught up with source: 2021-02-11T09:48:56.073 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000011.32.5, , 00000000004.11.5, , 00000000010.25.5, , , , , ] 2021-02-11T09:48:56.073 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000011.32.5, , 00000000004.11.5, , 00000000010.25.5, , , , , ] 2021-02-11T09:48:56.073 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:48:56.077+0000 7fa572d7c080 0 bucket checkpoint complete 2021-02-11T09:48:56.103 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:56.103 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:56.103 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:48:56.151 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:48:56.151 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:48:56.365 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:48:56.370+0000 7f15c3ef7080 1 bucket sync caught up with source: 2021-02-11T09:48:56.365 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, 00000000011.32.5, , 00000000004.11.5, , 00000000010.25.5, , , , , ] 2021-02-11T09:48:56.365 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, 00000000011.32.5, , 00000000004.11.5, , 00000000010.25.5, , , , , ] 2021-02-11T09:48:56.365 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:48:56.370+0000 7f15c3ef7080 0 bucket checkpoint complete 2021-02-11T09:48:56.394 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:56.394 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:56.394 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:48:56.441 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:48:56.441 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:48:56.648 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:48:56.653+0000 7fac2c2d4080 1 bucket sync caught up with source: 2021-02-11T09:48:56.648 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000004.17.5, , 00000000011.31.5, , 00000000008.26.5, , , , , ] 2021-02-11T09:48:56.648 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000004.17.5, , 00000000011.31.5, , 00000000008.26.5, , , , , ] 2021-02-11T09:48:56.648 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:48:56.653+0000 7fac2c2d4080 0 bucket checkpoint complete 2021-02-11T09:48:56.682 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:56.682 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:56.682 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:48:56.726 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:48:56.726 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:48:56.927 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:48:56.933+0000 7feb01dce080 1 bucket sync caught up with source: 2021-02-11T09:48:56.928 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000004.17.5, , 00000000011.31.5, , 00000000008.26.5, , , , , ] 2021-02-11T09:48:56.928 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000004.17.5, , 00000000011.31.5, , 00000000008.26.5, , , , , ] 2021-02-11T09:48:56.928 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:48:56.933+0000 7feb01dce080 0 bucket checkpoint complete 2021-02-11T09:48:56.959 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:56.959 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:56.959 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:48:57.051 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:48:57.051 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:48:57.244 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:48:57.249+0000 7fa172ab2080 1 bucket sync caught up with source: 2021-02-11T09:48:57.244 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000004.19.5, , 00000000004.15.5, , 00000000015.39.5, , , , , ] 2021-02-11T09:48:57.244 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000004.19.5, , 00000000004.15.5, , 00000000015.39.5, , , , , ] 2021-02-11T09:48:57.244 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:48:57.249+0000 7fa172ab2080 0 bucket checkpoint complete 2021-02-11T09:48:57.272 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:57.272 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:48:57.272 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:48:57.320 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:48:57.320 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:48:57.553 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:48:57.558+0000 7fb1bc231080 1 bucket sync caught up with source: 2021-02-11T09:48:57.553 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, 00000000004.19.5, , 00000000004.15.5, , 00000000015.39.5, , , , , ] 2021-02-11T09:48:57.553 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, 00000000004.19.5, , 00000000004.15.5, , 00000000015.39.5, , , , , ] 2021-02-11T09:48:57.553 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:48:57.558+0000 7fb1bc231080 0 bucket checkpoint complete 2021-02-11T09:48:57.586 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-16 zones={test-zone2, test-zone1} 2021-02-11T09:48:57.596 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:48:57.603 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:48:57.603 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T09:48:57.603 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T09:48:57.604 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T09:48:57.604 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:48:57.610 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:48:57.611 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T09:48:57.611 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T09:48:57.611 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T09:48:57.622 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:48:57.632 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:48:57.701 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:48:57.761 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:48:57.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:48:57.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:48:58.045 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:48:58.103 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:48:58.220 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:48:58.326 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:48:58.381 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:48:58.493 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:48:58.598 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-16 zones={test-zone2, test-zone1} 2021-02-11T09:48:58.598 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-16 zones={test-zone3, test-zone1} 2021-02-11T09:48:58.608 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:48:58.613 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:48:58.613 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T09:48:58.613 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T09:48:58.613 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T09:48:58.613 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:48:58.617 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:48:58.617 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T09:48:58.617 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T09:48:58.617 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T09:48:58.625 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:48:58.634 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:48:58.700 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:48:58.756 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:48:58.808 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:48:58.921 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:48:59.025 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:48:59.080 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:48:59.193 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:48:59.297 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:48:59.352 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:48:59.472 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:48:59.582 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-16 zones={test-zone3, test-zone1} 2021-02-11T09:48:59.582 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-16 zones={test-zone3, test-zone2} 2021-02-11T09:48:59.593 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:48:59.597 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:48:59.598 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T09:48:59.598 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T09:48:59.598 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T09:48:59.598 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:48:59.603 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:48:59.603 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T09:48:59.604 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T09:48:59.604 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T09:48:59.614 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:48:59.623 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:48:59.689 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:48:59.749 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:48:59.804 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:48:59.917 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:49:00.030 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:49:00.084 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:49:00.200 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:49:00.306 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:49:00.360 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:49:00.474 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:49:00.578 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-16 zones={test-zone3, test-zone2} 2021-02-11T09:49:00.578 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:49:00.578 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:49:00.579 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:49:00.622 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:00.622 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:00.827 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:49:00.830+0000 7fc1ba8b3080 1 bucket sync caught up with source: 2021-02-11T09:49:00.827 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, 00000000011.52.5, , 00000000004.13.5, , 00000000012.54.5, , , , , ] 2021-02-11T09:49:00.827 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, 00000000011.52.5, , 00000000004.13.5, , 00000000012.54.5, , , , , ] 2021-02-11T09:49:00.827 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:49:00.830+0000 7fc1ba8b3080 0 bucket checkpoint complete 2021-02-11T09:49:00.865 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:49:00.865 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:49:00.865 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:49:00.913 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:00.913 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:01.101 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:49:01.105+0000 7f7aec2e9080 1 bucket sync caught up with source: 2021-02-11T09:49:01.102 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000011.52.5, , 00000000004.13.5, , 00000000012.54.5, , , , , ] 2021-02-11T09:49:01.102 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000011.52.5, , 00000000004.13.5, , 00000000012.54.5, , , , , ] 2021-02-11T09:49:01.102 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:49:01.105+0000 7f7aec2e9080 0 bucket checkpoint complete 2021-02-11T09:49:01.128 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:49:01.128 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:49:01.129 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:49:01.176 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:01.176 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:01.380 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:49:01.382+0000 7ff227e32080 1 bucket sync caught up with source: 2021-02-11T09:49:01.380 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, 00000000011.52.5, , 00000000004.13.5, , 00000000012.54.5, , , , , ] 2021-02-11T09:49:01.380 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, 00000000011.52.5, , 00000000004.13.5, , 00000000012.54.5, , , , , ] 2021-02-11T09:49:01.380 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:49:01.382+0000 7ff227e32080 0 bucket checkpoint complete 2021-02-11T09:49:01.404 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:49:01.404 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:49:01.404 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:49:01.455 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:01.455 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:01.655 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:49:01.661+0000 7f5f8637a080 1 bucket sync caught up with source: 2021-02-11T09:49:01.655 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000004.46.5, , 00000000011.35.5, , 00000000013.44.5, , , , , ] 2021-02-11T09:49:01.656 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000004.46.5, , 00000000011.35.5, , 00000000013.44.5, , , , , ] 2021-02-11T09:49:01.656 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:49:01.661+0000 7f5f8637a080 0 bucket checkpoint complete 2021-02-11T09:49:01.686 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:49:01.686 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:49:01.686 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:49:01.731 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:01.731 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:01.929 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:49:01.933+0000 7feb1bba0080 1 bucket sync caught up with source: 2021-02-11T09:49:01.929 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000004.46.5, , 00000000011.35.5, , 00000000013.44.5, , , , , ] 2021-02-11T09:49:01.930 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000004.46.5, , 00000000011.35.5, , 00000000013.44.5, , , , , ] 2021-02-11T09:49:01.930 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:49:01.933+0000 7feb1bba0080 0 bucket checkpoint complete 2021-02-11T09:49:01.962 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:49:01.962 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:49:01.962 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:49:02.016 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:02.016 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:02.200 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:49:02.205+0000 7fcec0ed5080 1 bucket sync caught up with source: 2021-02-11T09:49:02.200 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000004.47.5, , 00000000004.17.5, , 00000000015.45.5, , , , , ] 2021-02-11T09:49:02.200 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000004.47.5, , 00000000004.17.5, , 00000000015.45.5, , , , , ] 2021-02-11T09:49:02.200 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:49:02.205+0000 7fcec0ed5080 0 bucket checkpoint complete 2021-02-11T09:49:02.226 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:49:02.226 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:49:02.227 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:49:02.274 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:02.274 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:02.496 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:49:02.498+0000 7f1708d41080 1 bucket sync caught up with source: 2021-02-11T09:49:02.496 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, 00000000004.47.5, , 00000000004.17.5, , 00000000015.45.5, , , , , ] 2021-02-11T09:49:02.496 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, 00000000004.47.5, , 00000000004.17.5, , 00000000015.45.5, , , , , ] 2021-02-11T09:49:02.496 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:49:02.498+0000 7f1708d41080 0 bucket checkpoint complete 2021-02-11T09:49:02.527 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-17 zones={test-zone2, test-zone1} 2021-02-11T09:49:02.536 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:49:02.542 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:49:02.542 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T09:49:02.542 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T09:49:02.542 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T09:49:02.542 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:49:02.547 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:49:02.548 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T09:49:02.548 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T09:49:02.548 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T09:49:02.558 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:49:02.568 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:49:02.636 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:49:02.697 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:49:02.752 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:49:02.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:49:02.970 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:49:03.024 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:49:03.144 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:49:03.245 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:49:03.297 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:49:03.410 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:49:03.513 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-17 zones={test-zone2, test-zone1} 2021-02-11T09:49:03.514 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-17 zones={test-zone3, test-zone1} 2021-02-11T09:49:03.522 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:49:03.526 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:49:03.526 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T09:49:03.526 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T09:49:03.526 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T09:49:03.527 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:49:03.530 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:49:03.531 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T09:49:03.531 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T09:49:03.531 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T09:49:03.539 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:49:03.547 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:49:03.610 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:49:03.669 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:49:03.724 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:49:03.837 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:49:03.941 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:49:03.996 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:49:04.108 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:49:04.212 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:49:04.269 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:49:04.381 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:49:04.485 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-17 zones={test-zone3, test-zone1} 2021-02-11T09:49:04.485 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-17 zones={test-zone3, test-zone2} 2021-02-11T09:49:04.495 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:49:04.500 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:49:04.500 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T09:49:04.500 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T09:49:04.500 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T09:49:04.500 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:49:04.551 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:49:04.551 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T09:49:04.551 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T09:49:04.551 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T09:49:04.606 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:49:04.618 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:49:04.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:49:04.741 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:49:04.796 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:49:04.908 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:49:05.017 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:49:05.072 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:49:05.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:49:05.289 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:49:05.343 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:49:05.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:49:05.557 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-17 zones={test-zone3, test-zone2} 2021-02-11T09:49:05.557 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:49:05.557 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:49:05.557 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:49:05.599 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:05.600 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:05.814 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:49:05.818+0000 7f30a777d080 1 bucket sync caught up with source: 2021-02-11T09:49:05.814 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, 00000000011.33.5, , 00000000004.14.5, , 00000000008.20.5, , , , , ] 2021-02-11T09:49:05.814 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, 00000000011.33.5, , 00000000004.14.5, , 00000000008.20.5, , , , , ] 2021-02-11T09:49:05.814 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:49:05.818+0000 7f30a777d080 0 bucket checkpoint complete 2021-02-11T09:49:05.845 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:49:05.846 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:49:05.846 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:49:05.888 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:05.889 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:06.063 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:49:06.069+0000 7fa0c2cb7080 1 bucket sync caught up with source: 2021-02-11T09:49:06.063 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000011.33.5, , 00000000004.14.5, , 00000000008.20.5, , , , , ] 2021-02-11T09:49:06.063 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000011.33.5, , 00000000004.14.5, , 00000000008.20.5, , , , , ] 2021-02-11T09:49:06.063 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:49:06.069+0000 7fa0c2cb7080 0 bucket checkpoint complete 2021-02-11T09:49:06.086 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:49:06.086 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:49:06.086 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:49:06.133 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:06.133 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:06.344 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:49:06.346+0000 7fb65cad2080 1 bucket sync caught up with source: 2021-02-11T09:49:06.344 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, 00000000011.33.5, , 00000000004.14.5, , 00000000008.20.5, , , , , ] 2021-02-11T09:49:06.344 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, 00000000011.33.5, , 00000000004.14.5, , 00000000008.20.5, , , , , ] 2021-02-11T09:49:06.345 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:49:06.346+0000 7fb65cad2080 0 bucket checkpoint complete 2021-02-11T09:49:06.377 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:49:06.377 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:49:06.377 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:49:06.423 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:06.423 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:06.628 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:49:06.633+0000 7f57ac793080 1 bucket sync caught up with source: 2021-02-11T09:49:06.629 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000004.18.5, , 00000000011.37.5, , 00000000009.25.5, , , , , ] 2021-02-11T09:49:06.629 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000004.18.5, , 00000000011.37.5, , 00000000009.25.5, , , , , ] 2021-02-11T09:49:06.629 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:49:06.633+0000 7f57ac793080 0 bucket checkpoint complete 2021-02-11T09:49:06.661 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:49:06.661 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:49:06.661 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:49:06.708 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:06.708 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:06.908 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:49:06.913+0000 7f1cc5649080 1 bucket sync caught up with source: 2021-02-11T09:49:06.908 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000004.18.5, , 00000000011.37.5, , 00000000009.25.5, , , , , ] 2021-02-11T09:49:06.909 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000004.18.5, , 00000000011.37.5, , 00000000009.25.5, , , , , ] 2021-02-11T09:49:06.909 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:49:06.913+0000 7f1cc5649080 0 bucket checkpoint complete 2021-02-11T09:49:06.939 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:49:06.939 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:49:06.939 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:49:06.986 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:06.986 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:07.190 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:49:07.193+0000 7f146196d080 1 bucket sync caught up with source: 2021-02-11T09:49:07.190 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000005.26.5, , 00000000004.18.5, , 00000000017.42.5, , , , , ] 2021-02-11T09:49:07.190 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000005.26.5, , 00000000004.18.5, , 00000000017.42.5, , , , , ] 2021-02-11T09:49:07.190 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:49:07.193+0000 7f146196d080 0 bucket checkpoint complete 2021-02-11T09:49:07.217 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:49:07.218 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:49:07.218 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:49:07.265 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:07.265 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:07.471 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:49:07.474+0000 7f6eddb1f080 1 bucket sync caught up with source: 2021-02-11T09:49:07.471 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, 00000000005.26.5, , 00000000004.18.5, , 00000000017.42.5, , , , , ] 2021-02-11T09:49:07.471 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, 00000000005.26.5, , 00000000004.18.5, , 00000000017.42.5, , , , , ] 2021-02-11T09:49:07.472 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:49:07.474+0000 7f6eddb1f080 0 bucket checkpoint complete 2021-02-11T09:49:07.503 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-18 zones={test-zone2, test-zone1} 2021-02-11T09:49:07.512 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:49:07.518 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:49:07.518 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T09:49:07.518 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T09:49:07.518 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T09:49:07.518 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:49:07.523 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:49:07.524 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T09:49:07.524 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T09:49:07.524 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T09:49:07.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:49:07.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:49:07.607 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:49:07.664 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:49:07.720 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:49:07.835 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:49:07.943 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:49:08.001 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:49:08.116 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:49:08.221 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:49:08.275 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:49:08.387 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:49:08.494 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-18 zones={test-zone2, test-zone1} 2021-02-11T09:49:08.494 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-18 zones={test-zone3, test-zone1} 2021-02-11T09:49:08.503 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:49:08.510 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:49:08.510 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T09:49:08.510 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T09:49:08.510 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T09:49:08.510 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:49:08.519 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:49:08.519 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T09:49:08.519 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T09:49:08.519 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T09:49:08.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:49:08.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:49:08.624 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:49:08.684 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:49:08.741 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:49:08.854 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:49:08.952 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:49:09.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:49:09.120 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:49:09.229 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:49:09.282 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:49:09.396 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:49:09.499 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-18 zones={test-zone3, test-zone1} 2021-02-11T09:49:09.499 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-18 zones={test-zone3, test-zone2} 2021-02-11T09:49:09.509 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:49:09.513 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:49:09.513 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T09:49:09.513 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T09:49:09.513 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T09:49:09.513 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:49:09.519 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:49:09.519 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T09:49:09.519 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T09:49:09.519 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T09:49:09.529 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:49:09.538 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:49:09.607 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:49:09.669 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:49:09.724 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:49:09.836 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T09:49:09.941 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:49:09.995 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:49:10.111 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T09:49:10.217 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:49:10.271 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:49:10.381 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T09:49:10.484 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-18 zones={test-zone3, test-zone2} 2021-02-11T09:49:10.484 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_versioned_object_incremental_sync ... ok 2021-02-11T09:49:10.525 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:10.525 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:10.526 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:49:10.559 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:10.559 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:10.718 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1 2021-02-11T09:49:10.752 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613036950.530761_232.1","last_update":"2021-02-11T09:49:10.530761Z"},{"marker":"1_1613036864.081584_81.1","last_update":"2021-02-11T09:47:44.081584Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T09:49:10.752 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T09:49:10.753 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:10.753 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:10.753 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:49:10.793 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:10.793 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:11.038 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036950.530761_232.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:10.530761Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:49:11.039 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T09:49:11.039 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T09:49:11.039 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:11.039 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:11.039 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:49:11.080 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:11.080 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:11.303 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036950.530761_232.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:10.530761Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:49:11.303 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T09:49:11.304 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T09:49:11.304 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:11.304 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:11.304 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:49:11.343 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:11.343 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:11.555 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036950.530761_232.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:10.530761Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:49:11.555 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T09:49:11.606 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:49:11.607 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:49:11.607 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:49:11.655 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:11.655 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:11.865 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:49:11.865+0000 7fac0325f080 1 bucket sync caught up with source: 2021-02-11T09:49:11.866 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, , , , , 00000000002.11.10, , , , , ] 2021-02-11T09:49:11.866 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, , , , , 00000000002.11.10, , , , , ] 2021-02-11T09:49:11.866 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:49:11.865+0000 7fac0325f080 0 bucket checkpoint complete 2021-02-11T09:49:11.895 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:49:11.895 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:49:11.896 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:49:11.949 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:11.949 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:12.105 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:49:12.108+0000 7fb7ac705080 1 waiting for incremental sync to catch up: 2021-02-11T09:49:12.105 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , 00000000002.11.10, , , , , ] 2021-02-11T09:49:12.105 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , 00000000005.17.9, , , , , ] 2021-02-11T09:49:17.109 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:49:17.111+0000 7fb7ac705080 1 bucket sync caught up with source: 2021-02-11T09:49:17.109 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , 00000000006.19.9, , , , , ] 2021-02-11T09:49:17.110 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , 00000000005.17.9, , , , , ] 2021-02-11T09:49:17.110 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:49:17.111+0000 7fb7ac705080 0 bucket checkpoint complete 2021-02-11T09:49:17.142 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:49:17.142 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:49:17.142 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:49:17.190 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:17.190 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:17.379 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:49:17.380+0000 7f07df955080 1 bucket sync caught up with source: 2021-02-11T09:49:17.379 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, , , , , 00000000006.19.9, , , , , ] 2021-02-11T09:49:17.379 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, , , , , 00000000006.19.9, , , , , ] 2021-02-11T09:49:17.380 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:49:17.380+0000 7f07df955080 0 bucket checkpoint complete 2021-02-11T09:49:17.409 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:49:17.409 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:49:17.409 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:49:17.456 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:17.456 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:17.664 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:49:17.663+0000 7ff2e83ce080 1 bucket sync caught up with source: 2021-02-11T09:49:17.664 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , 00000000004.23.9, , , , , ] 2021-02-11T09:49:17.664 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , 00000000004.23.9, , , , , ] 2021-02-11T09:49:17.664 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:49:17.663+0000 7ff2e83ce080 0 bucket checkpoint complete 2021-02-11T09:49:17.698 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:49:17.698 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:49:17.698 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:49:17.787 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:17.787 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:17.987 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:49:17.987+0000 7f23f74df080 1 bucket sync caught up with source: 2021-02-11T09:49:17.987 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , 00000000004.23.9, , , , , ] 2021-02-11T09:49:17.987 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , 00000000004.23.9, , , , , ] 2021-02-11T09:49:17.987 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:49:17.987+0000 7f23f74df080 0 bucket checkpoint complete 2021-02-11T09:49:18.023 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:49:18.023 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:49:18.023 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:49:18.068 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:18.069 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:18.280 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:49:18.283+0000 7f2258b56080 1 bucket sync caught up with source: 2021-02-11T09:49:18.280 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , 00000000004.21.9, , , , , ] 2021-02-11T09:49:18.280 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , 00000000004.21.9, , , , , ] 2021-02-11T09:49:18.280 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:49:18.283+0000 7f2258b56080 0 bucket checkpoint complete 2021-02-11T09:49:18.309 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:49:18.310 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:49:18.310 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:49:18.356 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:18.356 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:18.588 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:49:18.588+0000 7f9f909a9080 1 bucket sync caught up with source: 2021-02-11T09:49:18.588 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, , , , , 00000000004.21.9, , , , , ] 2021-02-11T09:49:18.588 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, , , , , 00000000004.21.9, , , , , ] 2021-02-11T09:49:18.588 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:49:18.588+0000 7f9f909a9080 0 bucket checkpoint complete 2021-02-11T09:49:18.621 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-19 zones={test-zone2, test-zone1} 2021-02-11T09:49:18.631 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:49:18.679 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:49:18.679 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:49:18.686 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:49:18.741 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:49:18.752 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:49:18.816 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:49:18.877 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-19 zones={test-zone2, test-zone1} 2021-02-11T09:49:18.877 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-19 zones={test-zone3, test-zone1} 2021-02-11T09:49:18.886 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:49:18.934 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:49:18.935 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:49:18.940 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:49:18.993 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:49:19.004 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:49:19.067 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:49:19.124 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-19 zones={test-zone3, test-zone1} 2021-02-11T09:49:19.124 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-19 zones={test-zone3, test-zone2} 2021-02-11T09:49:19.134 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:49:19.178 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:49:19.179 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:49:19.184 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:49:19.237 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:49:19.250 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:49:19.316 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:49:19.373 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-19 zones={test-zone3, test-zone2} 2021-02-11T09:49:20.834 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:49:20.835 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:49:20.835 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:49:20.884 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:20.884 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:21.085 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:49:21.084+0000 7f1f68ca2080 1 waiting for incremental sync to catch up: 2021-02-11T09:49:21.085 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, , , , , 00000000006.19.9, , , 00000000048.128.11, , ] 2021-02-11T09:49:21.086 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, , , , , 00000000006.19.9, , , 00000000049.130.9, , ] 2021-02-11T09:49:26.090 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:49:26.087+0000 7f1f68ca2080 1 bucket sync caught up with source: 2021-02-11T09:49:26.090 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, , , , , 00000000006.19.9, , , 00000000049.130.9, , ] 2021-02-11T09:49:26.091 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, , , , , 00000000006.19.9, , , 00000000049.130.9, , ] 2021-02-11T09:49:26.091 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:49:26.087+0000 7f1f68ca2080 0 bucket checkpoint complete 2021-02-11T09:49:26.124 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:49:26.124 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:49:26.124 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:49:26.171 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:26.171 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:26.358 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:49:26.362+0000 7fec3abbe080 1 bucket sync caught up with source: 2021-02-11T09:49:26.358 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , 00000000006.19.9, , , 00000000049.130.9, , ] 2021-02-11T09:49:26.358 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , 00000000006.19.9, , , 00000000049.130.9, , ] 2021-02-11T09:49:26.358 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:49:26.362+0000 7fec3abbe080 0 bucket checkpoint complete 2021-02-11T09:49:26.384 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:49:26.385 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:49:26.385 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:49:26.429 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:26.429 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:26.648 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:49:26.647+0000 7f6d05d5d080 1 bucket sync caught up with source: 2021-02-11T09:49:26.648 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, , , , , 00000000006.19.9, , , 00000000049.130.9, , ] 2021-02-11T09:49:26.648 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, , , , , 00000000006.19.9, , , 00000000049.130.9, , ] 2021-02-11T09:49:26.648 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:49:26.647+0000 7f6d05d5d080 0 bucket checkpoint complete 2021-02-11T09:49:26.685 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:49:26.685 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:49:26.685 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:49:26.733 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:26.733 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:26.932 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:49:26.933+0000 7fe81866d080 1 bucket sync caught up with source: 2021-02-11T09:49:26.933 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , 00000000004.23.9, , , 00000000056.140.9, , ] 2021-02-11T09:49:26.933 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , 00000000004.23.9, , , 00000000056.140.9, , ] 2021-02-11T09:49:26.933 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:49:26.933+0000 7fe81866d080 0 bucket checkpoint complete 2021-02-11T09:49:26.968 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:49:26.969 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:49:26.969 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:49:27.010 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:27.010 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:27.207 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:49:27.209+0000 7fa4b1868080 1 bucket sync caught up with source: 2021-02-11T09:49:27.208 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , 00000000004.23.9, , , 00000000056.140.9, , ] 2021-02-11T09:49:27.208 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , 00000000004.23.9, , , 00000000056.140.9, , ] 2021-02-11T09:49:27.208 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:49:27.209+0000 7fa4b1868080 0 bucket checkpoint complete 2021-02-11T09:49:27.238 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:49:27.239 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:49:27.239 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:49:27.327 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:27.327 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:27.519 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:49:27.521+0000 7f64ba820080 1 bucket sync caught up with source: 2021-02-11T09:49:27.519 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , 00000000004.21.9, , , 00000000058.143.5, , ] 2021-02-11T09:49:27.519 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , 00000000004.21.9, , , 00000000058.143.5, , ] 2021-02-11T09:49:27.520 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:49:27.521+0000 7f64ba820080 0 bucket checkpoint complete 2021-02-11T09:49:27.547 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:49:27.547 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:49:27.547 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:49:27.593 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:27.594 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:27.806 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:49:27.803+0000 7f92a642d080 1 bucket sync caught up with source: 2021-02-11T09:49:27.807 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, , , , , 00000000004.21.9, , , 00000000058.143.5, , ] 2021-02-11T09:49:27.808 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, , , , , 00000000004.21.9, , , 00000000058.143.5, , ] 2021-02-11T09:49:27.808 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:49:27.803+0000 7f92a642d080 0 bucket checkpoint complete 2021-02-11T09:49:27.841 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-19 zones={test-zone2, test-zone1} 2021-02-11T09:49:27.852 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:49:27.860 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:49:27.861 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:27.861 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:27.861 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:27.861 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:27.861 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:27.861 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:27.861 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:27.861 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:27.861 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:27.861 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:27.861 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:27.861 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:27.861 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:27.861 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:27.861 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:27.862 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:27.862 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:27.862 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:27.862 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:27.862 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:27.862 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:27.862 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:27.862 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:27.862 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:27.862 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:27.862 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:27.862 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:27.862 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:27.862 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:27.863 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:27.863 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:49:27.873 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:49:27.873 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:27.873 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:27.873 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:27.873 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:27.873 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:27.873 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:27.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:27.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:27.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:27.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:27.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:27.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:27.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:27.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:27.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:27.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:27.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:27.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:27.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:27.875 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:27.875 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:27.875 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:27.875 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:27.875 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:27.875 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:27.875 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:27.875 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:27.875 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:27.875 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:27.875 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:27.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:49:27.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:49:28.014 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:49:28.074 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:28.128 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:28.248 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:28.352 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:28.406 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:28.529 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:28.634 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:28.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:28.806 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:28.911 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:28.963 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:29.077 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:29.183 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:29.235 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:29.347 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:29.451 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:29.504 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:29.625 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:29.730 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:29.784 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:29.903 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:30.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:30.059 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:30.181 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:30.290 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:30.348 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:30.467 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:30.571 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:30.624 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:30.745 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:30.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:30.902 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:31.021 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:31.130 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:31.184 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:31.303 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:31.406 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:31.460 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:31.579 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:31.682 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:31.736 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:31.856 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:31.964 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:32.024 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:32.142 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:32.246 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:32.300 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:32.425 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:32.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:32.596 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:32.722 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:32.827 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:32.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:33.009 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:33.118 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:33.172 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:33.292 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:33.405 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:33.462 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:33.580 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:33.688 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:33.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:33.859 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:33.962 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:34.016 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:34.138 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:34.243 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:34.299 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:34.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:34.527 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:34.584 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:34.703 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:34.810 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:34.868 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:34.993 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:35.098 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:35.151 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:35.276 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:35.381 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:35.435 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:35.554 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:35.660 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:35.715 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:35.826 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:35.932 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:35.987 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:36.109 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:36.214 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:36.267 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:36.386 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:36.490 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-19 zones={test-zone2, test-zone1} 2021-02-11T09:49:36.490 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-19 zones={test-zone3, test-zone1} 2021-02-11T09:49:36.500 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:49:36.507 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:49:36.508 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:36.508 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:36.508 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:36.508 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:36.508 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:36.508 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:36.508 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:36.508 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:36.508 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:36.508 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:36.509 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:36.509 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:36.509 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:36.509 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:36.509 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:36.509 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:36.509 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:36.509 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:36.509 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:36.509 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:36.509 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:36.510 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:36.510 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:36.510 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:36.510 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:36.510 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:36.510 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:36.510 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:36.510 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:36.510 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:36.510 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:49:36.517 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:49:36.517 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:36.517 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:36.517 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:36.517 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:36.517 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:36.517 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:36.517 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:36.517 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:36.518 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:36.518 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:36.518 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:36.518 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:36.518 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:36.518 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:36.518 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:36.518 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:36.518 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:36.518 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:36.518 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:36.519 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:36.519 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:36.519 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:36.519 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:36.519 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:36.519 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:36.519 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:36.519 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:36.519 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:36.519 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:36.519 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:36.534 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:49:36.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:49:36.613 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:49:36.673 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:36.726 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:36.840 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:36.945 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:36.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:37.110 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:37.218 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:37.271 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:37.383 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:37.486 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:37.540 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:37.658 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:37.763 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:37.814 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:37.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:38.034 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:38.088 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:38.209 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:38.314 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:38.366 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:38.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:38.586 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:38.639 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:38.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:38.858 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:38.911 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:39.026 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:39.131 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:39.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:39.337 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:39.450 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:39.508 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:39.632 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:39.734 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:39.792 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:39.908 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:40.017 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:40.071 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:40.190 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:40.294 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:40.346 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:40.462 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:40.565 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:40.620 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:40.736 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:40.842 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:40.896 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:41.022 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:41.125 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:41.179 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:41.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:41.402 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:41.456 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:41.571 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:41.674 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:41.727 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:41.844 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:41.949 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:42.004 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:42.121 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:42.235 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:42.292 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:42.402 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:42.505 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:42.559 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:42.670 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:42.774 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:42.827 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:42.945 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:43.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:43.103 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:43.221 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:43.324 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:43.379 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:43.498 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:43.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:43.660 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:43.782 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:43.886 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:43.940 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:44.058 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:44.166 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:44.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:44.343 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:44.471 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:44.534 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:44.662 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:44.779 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:44.838 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:44.961 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:45.067 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-19 zones={test-zone3, test-zone1} 2021-02-11T09:49:45.067 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-19 zones={test-zone3, test-zone2} 2021-02-11T09:49:45.080 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:49:45.087 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:49:45.087 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:45.087 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:45.087 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:45.087 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:45.087 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:45.087 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:45.087 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:45.088 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:45.088 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:45.088 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:45.088 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:45.088 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:45.088 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:45.088 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:45.088 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:45.088 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:45.088 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:45.088 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:45.088 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:45.088 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:45.088 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:45.089 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:45.089 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:45.089 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:45.089 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:45.089 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:45.089 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:45.089 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:45.089 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:45.089 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:45.089 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:49:45.102 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T09:49:45.102 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:45.103 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:45.103 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:45.103 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:45.103 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:45.103 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:45.103 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:45.103 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:45.103 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:45.103 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:45.103 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:45.103 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:45.103 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:45.103 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:45.103 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:45.103 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:45.104 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:45.104 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:45.104 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:45.104 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:45.104 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:45.104 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:45.104 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:45.104 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:45.104 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:45.104 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:45.104 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:45.104 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:45.104 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:45.104 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:49:45.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:49:45.135 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:49:45.207 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T09:49:45.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:45.327 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:45.443 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:45.546 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:45.604 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:45.724 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:45.830 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:45.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:46.004 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:46.111 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:46.165 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:46.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:46.394 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:46.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:46.568 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:46.674 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:46.728 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:46.849 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:46.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:47.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:47.128 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:47.235 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:47.292 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:47.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:47.523 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:47.580 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:47.700 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:47.807 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:47.859 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:47.979 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:48.087 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:48.143 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:48.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:48.371 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:48.428 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:48.547 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:48.655 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:48.715 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:48.837 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:48.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:49.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:49.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:49.230 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:49.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:49.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:49.500 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:49.555 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:49.669 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:50.324 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:50.379 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:50.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:50.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:50.655 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:50.771 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:50.874 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:50.928 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:51.051 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:51.158 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:51.216 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:51.333 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:51.438 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:51.491 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:51.607 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:51.710 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:51.763 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:51.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:51.990 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:52.043 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:52.163 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:52.271 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:52.323 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:52.445 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:52.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:52.604 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:52.722 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:52.826 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:52.880 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:53.004 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:53.110 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:53.164 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:53.286 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:53.389 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:53.443 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:53.560 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:53.667 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:53.720 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:53.840 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:53.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:54.004 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:54.124 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:49:54.231 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-19 zones={test-zone3, test-zone2} 2021-02-11T09:49:54.232 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:54.232 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:54.232 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:49:54.273 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:54.273 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:54.485 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613036964.844780_237.1","last_update":"2021-02-11T09:49:24.844780Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1613036942.835890_67.3","last_update":"2021-02-11T09:49:02.835890Z"},{"marker":"1_1613036964.844780_79.1","last_update":"2021-02-11T09:49:24.844780Z"}] 2021-02-11T09:49:54.486 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-02-11T09:49:54.486 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:54.486 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:54.486 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:49:54.528 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:54.528 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:54.753 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4703921989677143155},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613036964.844780_237.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:49:24.844780Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613036942.835890_67.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:49:02.835890Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613036964.844780_79.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:49:24.844780Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T09:49:54.753 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-02-11T09:49:54.753 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:54.753 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:54.754 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:49:54.792 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:54.792 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:55.012 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613036964.844780_237.1","last_update":"2021-02-11T09:49:24.844780Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1613036942.835890_67.3","last_update":"2021-02-11T09:49:02.835890Z"},{"marker":"1_1613036964.844780_79.1","last_update":"2021-02-11T09:49:24.844780Z"}] 2021-02-11T09:49:55.012 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-02-11T09:49:55.012 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:55.012 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:55.012 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:49:55.098 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:55.098 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:55.317 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":13312279424886386302},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613036964.844780_237.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:49:24.844780Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613036942.835890_67.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:49:02.835890Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613036964.844780_79.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:49:24.844780Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T09:49:55.317 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-02-11T09:49:55.318 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:55.318 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:55.318 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:49:55.399 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:55.399 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:55.620 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613036964.844780_237.1","last_update":"2021-02-11T09:49:24.844780Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1613036942.835890_67.3","last_update":"2021-02-11T09:49:02.835890Z"},{"marker":"1_1613036964.844780_79.1","last_update":"2021-02-11T09:49:24.844780Z"}] 2021-02-11T09:49:55.621 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-02-11T09:49:55.621 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:55.621 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:55.621 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:49:55.662 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:55.662 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:55.910 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2896161496309628191},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613036964.844780_237.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:49:24.844780Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613036942.835890_67.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:49:02.835890Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613036964.844780_79.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:49:24.844780Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T09:49:55.910 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-02-11T09:49:55.910 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:55.911 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:55.911 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:49:55.990 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:55.991 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:56.233 INFO:teuthology.orchestra.run.smithi098.stdout:[{"marker":"1_1613036964.822510_109.1","last_update":"2021-02-11T09:49:24.822510Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1613036942.814064_78.3","last_update":"2021-02-11T09:49:02.814064Z"},{"marker":"1_1613036964.822510_88.1","last_update":"2021-02-11T09:49:24.822510Z"}] 2021-02-11T09:49:56.233 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-02-11T09:49:56.233 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:56.233 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:56.233 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:49:56.276 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:56.276 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:56.489 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":15226599365335882107},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613036964.822510_109.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:49:24.822510Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613036942.814064_78.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:49:02.814064Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613036964.822510_88.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:49:24.822510Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T09:49:56.490 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-02-11T09:49:56.490 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:56.490 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:56.490 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:49:56.529 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:56.530 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:56.770 INFO:teuthology.orchestra.run.smithi098.stdout:[{"marker":"1_1613036964.822510_109.1","last_update":"2021-02-11T09:49:24.822510Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1613036942.814064_78.3","last_update":"2021-02-11T09:49:02.814064Z"},{"marker":"1_1613036964.822510_88.1","last_update":"2021-02-11T09:49:24.822510Z"}] 2021-02-11T09:49:56.770 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-02-11T09:49:56.770 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:56.770 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:56.771 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:49:56.812 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:56.812 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:57.039 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":10999913493642558030},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613036964.822510_109.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:49:24.822510Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613036942.814064_78.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:49:02.814064Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613036964.822510_88.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:49:24.822510Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T09:49:57.039 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-02-11T09:49:57.039 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:57.040 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:57.040 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:49:57.120 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:57.120 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:57.329 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613036964.875382_129.1","last_update":"2021-02-11T09:49:24.875382Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1613036942.869312_60.3","last_update":"2021-02-11T09:49:02.869312Z"},{"marker":"1_1613036964.875382_56.1","last_update":"2021-02-11T09:49:24.875382Z"}] 2021-02-11T09:49:57.330 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-02-11T09:49:57.330 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:57.330 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:57.330 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:49:57.414 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:57.414 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:57.647 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2753764877348348355},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613036964.875382_129.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:49:24.875382Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613036942.869312_60.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:49:02.869312Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613036964.875382_56.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:49:24.875382Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T09:49:57.647 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-02-11T09:49:57.647 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:57.648 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:57.648 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:49:57.727 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:57.727 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:57.947 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613036964.875382_129.1","last_update":"2021-02-11T09:49:24.875382Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1613036942.869312_60.3","last_update":"2021-02-11T09:49:02.869312Z"},{"marker":"1_1613036964.875382_56.1","last_update":"2021-02-11T09:49:24.875382Z"}] 2021-02-11T09:49:57.948 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-02-11T09:49:57.948 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:57.948 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:57.948 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:49:57.993 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:57.993 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:58.248 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":7143279782539285748},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613036964.875382_129.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:49:24.875382Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613036942.869312_60.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:49:02.869312Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613036964.875382_56.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:49:24.875382Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T09:49:58.249 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-02-11T09:49:58.249 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_concurrent_versioned_object_incremental_sync ... ok 2021-02-11T09:49:58.282 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:58.282 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:58.282 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:49:58.317 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:58.317 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:58.483 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1 2021-02-11T09:49:58.518 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613036998.280111_241.1","last_update":"2021-02-11T09:49:58.280111Z"},{"marker":"1_1613036864.081584_81.1","last_update":"2021-02-11T09:47:44.081584Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T09:49:58.518 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T09:49:58.518 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:58.518 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:58.519 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:49:58.551 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:58.551 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:58.775 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036998.280111_241.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:58.280111Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:49:58.776 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T09:49:58.776 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T09:49:58.776 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:58.776 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:58.776 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:49:58.817 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:58.817 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:59.036 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036998.280111_241.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:58.280111Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:49:59.036 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T09:49:59.036 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T09:49:59.036 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:59.036 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:49:59.036 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:49:59.077 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:59.077 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:59.308 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613036998.280111_241.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:49:58.280111Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:49:59.308 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T09:49:59.354 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:49:59.355 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:49:59.355 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:49:59.405 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:49:59.405 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:49:59.612 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:49:59.608+0000 7f3ce3b74080 1 waiting for incremental sync to catch up: 2021-02-11T09:49:59.613 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, , , , , , , , , , ] 2021-02-11T09:49:59.613 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, , , , , , , , 00000000001.17.5, , ] 2021-02-11T09:50:04.617 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:50:04.612+0000 7f3ce3b74080 1 bucket sync caught up with source: 2021-02-11T09:50:04.617 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, , , , , , , , 00000000001.17.5, , ] 2021-02-11T09:50:04.617 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, , , , , , , , 00000000001.17.5, , ] 2021-02-11T09:50:04.617 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:50:04.612+0000 7f3ce3b74080 0 bucket checkpoint complete 2021-02-11T09:50:04.651 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:04.651 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:04.652 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:04.699 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:04.699 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:04.902 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:50:04.898+0000 7f156a18d080 1 bucket sync caught up with source: 2021-02-11T09:50:04.902 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , 00000000001.17.5, , ] 2021-02-11T09:50:04.902 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , 00000000001.17.5, , ] 2021-02-11T09:50:04.902 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:50:04.898+0000 7f156a18d080 0 bucket checkpoint complete 2021-02-11T09:50:04.928 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:04.928 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:04.929 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:04.976 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:04.976 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:05.200 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:50:05.196+0000 7f433c05e080 1 bucket sync caught up with source: 2021-02-11T09:50:05.200 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, , , , , , , , 00000000001.17.5, , ] 2021-02-11T09:50:05.200 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, , , , , , , , 00000000001.17.5, , ] 2021-02-11T09:50:05.200 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:50:05.196+0000 7f433c05e080 0 bucket checkpoint complete 2021-02-11T09:50:05.232 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:05.232 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:05.233 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:05.281 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:05.281 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:05.484 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:50:05.478+0000 7f90e711d080 1 bucket sync caught up with source: 2021-02-11T09:50:05.484 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , 00000000001.40.5, , ] 2021-02-11T09:50:05.484 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , 00000000001.40.5, , ] 2021-02-11T09:50:05.484 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:50:05.478+0000 7f90e711d080 0 bucket checkpoint complete 2021-02-11T09:50:05.520 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:05.520 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:05.520 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:05.567 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:05.567 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:05.774 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:50:05.770+0000 7f5898d32080 1 bucket sync caught up with source: 2021-02-11T09:50:05.774 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , 00000000001.40.5, , ] 2021-02-11T09:50:05.774 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , 00000000001.40.5, , ] 2021-02-11T09:50:05.774 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:50:05.770+0000 7f5898d32080 0 bucket checkpoint complete 2021-02-11T09:50:05.807 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:05.807 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:05.807 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:05.853 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:05.853 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:06.056 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:50:06.050+0000 7f10523f5080 1 bucket sync caught up with source: 2021-02-11T09:50:06.056 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , 00000000001.21.5, , ] 2021-02-11T09:50:06.056 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , 00000000001.21.5, , ] 2021-02-11T09:50:06.056 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:50:06.050+0000 7f10523f5080 0 bucket checkpoint complete 2021-02-11T09:50:06.083 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:06.083 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:06.083 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:06.132 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:06.132 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:06.344 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:50:06.340+0000 7fa9d300b080 1 bucket sync caught up with source: 2021-02-11T09:50:06.345 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, , , , , , , , 00000000001.21.5, , ] 2021-02-11T09:50:06.345 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, , , , , , , , 00000000001.21.5, , ] 2021-02-11T09:50:06.345 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:50:06.340+0000 7fa9d300b080 0 bucket checkpoint complete 2021-02-11T09:50:06.377 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-20 zones={test-zone2, test-zone1} 2021-02-11T09:50:06.386 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:50:06.434 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:06.435 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:50:06.441 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:06.497 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:06.508 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:06.575 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:06.637 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-20 zones={test-zone2, test-zone1} 2021-02-11T09:50:06.637 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-20 zones={test-zone3, test-zone1} 2021-02-11T09:50:06.646 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:50:06.690 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:06.691 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:50:06.696 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:06.749 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:06.759 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:06.822 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:06.880 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-20 zones={test-zone3, test-zone1} 2021-02-11T09:50:06.881 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-20 zones={test-zone3, test-zone2} 2021-02-11T09:50:06.891 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:50:06.938 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:06.939 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:50:06.946 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:07.001 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:07.011 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:07.074 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:07.137 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-20 zones={test-zone3, test-zone2} 2021-02-11T09:50:07.152 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:50:07.152 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:50:07.152 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:50:07.191 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:07.191 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:08.096 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1 2021-02-11T09:50:08.129 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613037007.150589_244.1","last_update":"2021-02-11T09:50:07.150589Z"},{"marker":"1_1613036864.081584_81.1","last_update":"2021-02-11T09:47:44.081584Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T09:50:08.129 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T09:50:08.129 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:50:08.130 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:50:08.130 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:50:08.172 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:08.172 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:08.424 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037007.150589_244.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:50:07.150589Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:50:08.424 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T09:50:08.424 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T09:50:08.424 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:50:08.425 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:50:08.425 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:50:08.466 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:08.466 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:08.691 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037007.150589_244.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:50:07.150589Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:50:08.691 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T09:50:08.691 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T09:50:08.691 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:50:08.691 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:50:08.692 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:50:08.733 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:08.733 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:08.979 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037007.150589_244.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:50:07.150589Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:50:08.980 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T09:50:09.030 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:09.031 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:09.031 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:09.081 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:09.081 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:09.300 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:50:09.296+0000 7f205a7ef080 1 bucket sync caught up with source: 2021-02-11T09:50:09.300 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, , , , , , , , 00000000003.20.12, , ] 2021-02-11T09:50:09.300 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, , , , , , , , 00000000003.20.12, , ] 2021-02-11T09:50:09.300 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:50:09.296+0000 7f205a7ef080 0 bucket checkpoint complete 2021-02-11T09:50:09.335 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:09.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:09.335 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:09.381 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:09.381 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:09.568 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:50:09.566+0000 7f1872f97080 1 bucket sync caught up with source: 2021-02-11T09:50:09.568 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , 00000000003.20.12, , ] 2021-02-11T09:50:09.569 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , 00000000003.20.12, , ] 2021-02-11T09:50:09.569 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:50:09.566+0000 7f1872f97080 0 bucket checkpoint complete 2021-02-11T09:50:09.596 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:09.597 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:09.597 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:09.643 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:09.643 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:09.858 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:50:09.856+0000 7f3b87eb8080 1 bucket sync caught up with source: 2021-02-11T09:50:09.858 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, , , , , , , , 00000000003.20.12, , ] 2021-02-11T09:50:09.858 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, , , , , , , , 00000000003.20.12, , ] 2021-02-11T09:50:09.858 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:50:09.856+0000 7f3b87eb8080 0 bucket checkpoint complete 2021-02-11T09:50:09.891 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:09.892 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:09.892 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:09.941 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:09.941 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:10.164 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:50:10.162+0000 7fbf712dd080 1 bucket sync caught up with source: 2021-02-11T09:50:10.165 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , 00000000004.46.9, , ] 2021-02-11T09:50:10.165 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , 00000000004.46.9, , ] 2021-02-11T09:50:10.165 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:50:10.162+0000 7fbf712dd080 0 bucket checkpoint complete 2021-02-11T09:50:10.196 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:10.196 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:10.196 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:10.250 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:10.251 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:10.448 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:50:10.446+0000 7fe01cf24080 1 bucket sync caught up with source: 2021-02-11T09:50:10.448 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , 00000000004.46.9, , ] 2021-02-11T09:50:10.448 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , 00000000004.46.9, , ] 2021-02-11T09:50:10.448 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:50:10.446+0000 7fe01cf24080 0 bucket checkpoint complete 2021-02-11T09:50:10.480 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:10.481 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:10.481 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:10.572 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:10.572 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:10.781 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:50:10.782+0000 7f1b723b9080 1 bucket sync caught up with source: 2021-02-11T09:50:10.781 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , 00000000004.27.9, , ] 2021-02-11T09:50:10.781 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , 00000000004.27.9, , ] 2021-02-11T09:50:10.782 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:50:10.782+0000 7f1b723b9080 0 bucket checkpoint complete 2021-02-11T09:50:10.811 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:10.811 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:10.811 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:10.858 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:10.858 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:11.088 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:50:11.084+0000 7f0ed151f080 1 bucket sync caught up with source: 2021-02-11T09:50:11.088 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, , , , , , , , 00000000004.27.9, , ] 2021-02-11T09:50:11.088 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, , , , , , , , 00000000004.27.9, , ] 2021-02-11T09:50:11.088 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:50:11.084+0000 7f0ed151f080 0 bucket checkpoint complete 2021-02-11T09:50:11.122 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-20 zones={test-zone2, test-zone1} 2021-02-11T09:50:11.132 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:50:11.183 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:11.183 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:11.183 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:50:11.189 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:11.190 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:11.244 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:11.257 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:11.322 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:11.381 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:11.392 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:11.459 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:11.521 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-20 zones={test-zone2, test-zone1} 2021-02-11T09:50:11.521 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-20 zones={test-zone3, test-zone1} 2021-02-11T09:50:11.531 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:50:11.578 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:11.579 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:11.579 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:50:11.584 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:11.584 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:11.637 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:11.648 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:11.712 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:11.768 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:11.778 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:11.843 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:11.901 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-20 zones={test-zone3, test-zone1} 2021-02-11T09:50:11.902 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-20 zones={test-zone3, test-zone2} 2021-02-11T09:50:11.913 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:50:11.959 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:11.959 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:11.959 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:50:11.966 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:11.966 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:12.021 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:12.032 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:12.099 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:12.161 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:12.172 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:12.241 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:12.301 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-20 zones={test-zone3, test-zone2} 2021-02-11T09:50:12.318 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:50:12.318 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:50:12.318 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:50:12.353 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:12.353 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:12.527 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1 2021-02-11T09:50:12.556 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613037012.316265_246.1","last_update":"2021-02-11T09:50:12.316265Z"},{"marker":"1_1613036864.081584_81.1","last_update":"2021-02-11T09:47:44.081584Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T09:50:12.557 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T09:50:12.557 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:50:12.557 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:50:12.557 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:50:12.595 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:12.595 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:12.830 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037012.316265_246.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:50:12.316265Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:50:12.831 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T09:50:12.831 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T09:50:12.831 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:50:12.831 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:50:12.831 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:50:12.871 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:12.871 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:13.090 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037012.316265_246.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:50:12.316265Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:50:13.090 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T09:50:13.090 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T09:50:13.091 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:50:13.091 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:50:13.091 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:50:13.132 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:13.133 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:13.360 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037012.316265_246.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:50:12.316265Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:50:13.361 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T09:50:13.410 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:13.411 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:13.411 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:13.458 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:13.459 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:13.662 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:50:13.660+0000 7fdc394a5080 1 bucket sync caught up with source: 2021-02-11T09:50:13.662 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, , , , , , , , 00000000005.24.17, , ] 2021-02-11T09:50:13.662 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, , , , , , , , 00000000005.24.17, , ] 2021-02-11T09:50:13.662 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:50:13.660+0000 7fdc394a5080 0 bucket checkpoint complete 2021-02-11T09:50:13.700 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:13.700 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:13.700 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:13.744 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:13.744 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:13.941 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:50:13.941+0000 7f1cd9912080 1 bucket sync caught up with source: 2021-02-11T09:50:13.941 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , 00000000005.24.17, , ] 2021-02-11T09:50:13.941 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , 00000000005.24.17, , ] 2021-02-11T09:50:13.941 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:50:13.941+0000 7f1cd9912080 0 bucket checkpoint complete 2021-02-11T09:50:13.966 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:13.967 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:13.967 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:14.015 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:14.015 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:14.242 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:50:14.236+0000 7f76a005b080 1 bucket sync caught up with source: 2021-02-11T09:50:14.242 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, , , , , , , , 00000000005.24.17, , ] 2021-02-11T09:50:14.242 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, , , , , , , , 00000000005.24.17, , ] 2021-02-11T09:50:14.242 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:50:14.236+0000 7f76a005b080 0 bucket checkpoint complete 2021-02-11T09:50:14.274 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:14.275 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:14.275 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:14.322 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:14.322 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:14.517 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:50:14.517+0000 7fe53efbf080 1 bucket sync caught up with source: 2021-02-11T09:50:14.517 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , 00000000007.53.14, , ] 2021-02-11T09:50:14.517 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , 00000000007.53.14, , ] 2021-02-11T09:50:14.517 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:50:14.517+0000 7fe53efbf080 0 bucket checkpoint complete 2021-02-11T09:50:14.552 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:14.552 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:14.552 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:14.597 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:14.597 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:14.801 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:50:14.801+0000 7f9920d0d080 1 bucket sync caught up with source: 2021-02-11T09:50:14.801 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , 00000000007.53.14, , ] 2021-02-11T09:50:14.801 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , 00000000007.53.14, , ] 2021-02-11T09:50:14.801 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:50:14.801+0000 7f9920d0d080 0 bucket checkpoint complete 2021-02-11T09:50:14.832 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:14.832 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:14.832 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:14.879 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:14.879 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:15.081 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:50:15.081+0000 7f6eb5294080 1 bucket sync caught up with source: 2021-02-11T09:50:15.081 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , 00000000008.38.14, , ] 2021-02-11T09:50:15.081 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , 00000000008.38.14, , ] 2021-02-11T09:50:15.082 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:50:15.081+0000 7f6eb5294080 0 bucket checkpoint complete 2021-02-11T09:50:15.110 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:15.111 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:15.111 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:15.158 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:15.158 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:15.399 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:50:15.396+0000 7f5ab38c2080 1 bucket sync caught up with source: 2021-02-11T09:50:15.400 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, , , , , , , , 00000000008.38.14, , ] 2021-02-11T09:50:15.400 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, , , , , , , , 00000000008.38.14, , ] 2021-02-11T09:50:15.400 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:50:15.396+0000 7f5ab38c2080 0 bucket checkpoint complete 2021-02-11T09:50:15.432 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-20 zones={test-zone2, test-zone1} 2021-02-11T09:50:15.454 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:50:15.526 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:15.526 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:15.526 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:50:15.533 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:15.534 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:15.614 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:15.651 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:15.745 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:15.816 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:15.830 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:15.902 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:15.967 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-20 zones={test-zone2, test-zone1} 2021-02-11T09:50:15.967 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-20 zones={test-zone3, test-zone1} 2021-02-11T09:50:15.976 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:50:16.023 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:16.023 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:16.023 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:50:16.028 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:16.028 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:16.085 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:16.096 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:16.159 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:16.216 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:16.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:16.289 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:16.349 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-20 zones={test-zone3, test-zone1} 2021-02-11T09:50:16.349 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-20 zones={test-zone3, test-zone2} 2021-02-11T09:50:16.359 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:50:16.406 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:16.407 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:16.407 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:50:16.412 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:16.412 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:16.466 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:16.478 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:16.547 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:16.609 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:16.619 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:16.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:16.749 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-20 zones={test-zone3, test-zone2} 2021-02-11T09:50:16.749 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_version_suspended_incremental_sync ... ok 2021-02-11T09:50:16.750 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=tkscyf-21 2021-02-11T09:50:16.782 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=tkscyf-22 2021-02-11T09:50:16.839 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=tkscyf-23 2021-02-11T09:50:16.935 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:50:16.935 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:50:16.935 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:50:16.968 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:16.968 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:17.135 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1 2021-02-11T09:50:17.162 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613037016.931312_264.1","last_update":"2021-02-11T09:50:16.931312Z"},{"marker":"1_1613036864.081584_81.1","last_update":"2021-02-11T09:47:44.081584Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T09:50:17.162 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T09:50:17.162 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:50:17.162 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:50:17.163 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:50:17.201 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:17.201 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:17.440 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037016.931312_264.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:50:16.931312Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:50:17.441 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T09:50:17.441 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T09:50:17.441 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:50:17.441 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:50:17.441 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:50:17.483 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:17.483 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:17.702 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037016.931312_264.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:50:16.931312Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:50:17.702 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T09:50:17.702 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T09:50:17.702 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:50:17.702 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:50:17.702 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:50:17.742 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:17.742 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:17.965 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037016.931312_264.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:50:16.931312Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:50:17.966 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T09:50:18.180 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:18.180 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:18.180 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:18.231 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:18.232 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:18.431 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:50:18.428+0000 7fb2769ba080 1 bucket sync caught up with source: 2021-02-11T09:50:18.431 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, , , , , , , , 00000000003.18.11, , ] 2021-02-11T09:50:18.431 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, , , , , , , , 00000000003.18.11, , ] 2021-02-11T09:50:18.432 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:50:18.428+0000 7fb2769ba080 0 bucket checkpoint complete 2021-02-11T09:50:18.464 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:18.464 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:18.465 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:18.514 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:18.514 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:18.676 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:50:18.673+0000 7f62b235b080 1 bucket sync caught up with source: 2021-02-11T09:50:18.676 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , 00000000003.18.11, , ] 2021-02-11T09:50:18.676 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , 00000000003.18.11, , ] 2021-02-11T09:50:18.676 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:50:18.673+0000 7f62b235b080 0 bucket checkpoint complete 2021-02-11T09:50:18.703 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:18.703 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:18.703 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:18.749 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:18.750 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:18.965 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:50:18.960+0000 7f76ea3d3080 1 bucket sync caught up with source: 2021-02-11T09:50:18.965 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, , , , , , , , 00000000003.18.11, , ] 2021-02-11T09:50:18.965 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, , , , , , , , 00000000003.18.11, , ] 2021-02-11T09:50:18.965 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:50:18.960+0000 7f76ea3d3080 0 bucket checkpoint complete 2021-02-11T09:50:19.000 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:19.000 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:19.000 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-21 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:19.047 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:19.047 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:19.258 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:50:19.257+0000 7fd27c26d080 1 bucket sync caught up with source: 2021-02-11T09:50:19.258 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , 00000000002.28.5, , ] 2021-02-11T09:50:19.258 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , 00000000002.28.5, , ] 2021-02-11T09:50:19.258 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:50:19.257+0000 7fd27c26d080 0 bucket checkpoint complete 2021-02-11T09:50:19.291 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:19.291 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:19.291 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-21 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:19.337 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:19.337 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:19.539 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:50:19.537+0000 7f8a14b5c080 1 bucket sync caught up with source: 2021-02-11T09:50:19.539 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , 00000000002.28.5, , ] 2021-02-11T09:50:19.539 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , 00000000002.28.5, , ] 2021-02-11T09:50:19.539 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:50:19.537+0000 7f8a14b5c080 0 bucket checkpoint complete 2021-02-11T09:50:19.571 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:19.571 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:19.571 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:19.617 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:19.617 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:19.811 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:50:19.809+0000 7fd860d86080 1 bucket sync caught up with source: 2021-02-11T09:50:19.811 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , 00000000002.28.5, , ] 2021-02-11T09:50:19.811 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , 00000000002.28.5, , ] 2021-02-11T09:50:19.811 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:50:19.809+0000 7fd860d86080 0 bucket checkpoint complete 2021-02-11T09:50:19.839 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:19.839 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:19.839 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:19.888 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:19.888 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:20.110 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:50:20.107+0000 7fb26127b080 1 bucket sync caught up with source: 2021-02-11T09:50:20.111 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, , , , , , , , 00000000002.28.5, , ] 2021-02-11T09:50:20.111 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, , , , , , , , 00000000002.28.5, , ] 2021-02-11T09:50:20.111 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:50:20.107+0000 7fb26127b080 0 bucket checkpoint complete 2021-02-11T09:50:20.143 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-21 zones={test-zone2, test-zone1} 2021-02-11T09:50:20.153 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:50:20.203 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:20.203 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:20.203 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:50:20.210 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:20.211 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:20.266 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:20.273 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:20.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:20.349 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-21 zones={test-zone2, test-zone1} 2021-02-11T09:50:20.349 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-21 zones={test-zone3, test-zone1} 2021-02-11T09:50:20.358 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:50:20.406 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:20.407 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:20.407 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:50:20.413 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:20.413 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:20.468 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:20.476 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:20.484 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:20.549 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-21 zones={test-zone3, test-zone1} 2021-02-11T09:50:20.549 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-21 zones={test-zone3, test-zone2} 2021-02-11T09:50:20.559 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:50:20.607 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:20.607 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:20.607 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:50:20.614 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:20.615 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:20.673 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:29.833 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:29.845 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:29.913 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-21 zones={test-zone3, test-zone2} 2021-02-11T09:50:29.913 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:29.913 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:29.914 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:29.960 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:29.960 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:30.195 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:50:30.191+0000 7f404742f080 1 bucket sync caught up with source: 2021-02-11T09:50:30.195 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, , , , , , , , 00000000002.11.5, , ] 2021-02-11T09:50:30.195 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, , , , , , , , 00000000002.11.5, , ] 2021-02-11T09:50:30.195 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:50:30.191+0000 7f404742f080 0 bucket checkpoint complete 2021-02-11T09:50:30.226 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:30.226 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:30.227 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:30.273 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:30.274 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:30.470 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:50:30.469+0000 7f74991c9080 1 bucket sync caught up with source: 2021-02-11T09:50:30.470 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , 00000000002.11.5, , ] 2021-02-11T09:50:30.470 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , 00000000002.11.5, , ] 2021-02-11T09:50:30.470 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:50:30.469+0000 7f74991c9080 0 bucket checkpoint complete 2021-02-11T09:50:30.499 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:30.499 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:30.499 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:30.544 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:30.544 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:30.760 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:50:30.755+0000 7f2e70e70080 1 bucket sync caught up with source: 2021-02-11T09:50:30.761 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, , , , , , , , 00000000002.11.5, , ] 2021-02-11T09:50:30.761 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, , , , , , , , 00000000002.11.5, , ] 2021-02-11T09:50:30.761 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:50:30.755+0000 7f2e70e70080 0 bucket checkpoint complete 2021-02-11T09:50:30.795 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:30.795 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:30.795 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-22 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:30.843 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:30.843 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:31.040 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:50:31.037+0000 7ff30a978080 1 bucket sync caught up with source: 2021-02-11T09:50:31.041 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , 00000000003.17.11, , ] 2021-02-11T09:50:31.041 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , 00000000003.17.11, , ] 2021-02-11T09:50:31.041 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:50:31.037+0000 7ff30a978080 0 bucket checkpoint complete 2021-02-11T09:50:31.072 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:31.073 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:31.073 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-22 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:31.117 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:31.117 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:31.321 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:50:31.317+0000 7fd893c9b080 1 bucket sync caught up with source: 2021-02-11T09:50:31.321 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , 00000000003.17.11, , ] 2021-02-11T09:50:31.321 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , 00000000003.17.11, , ] 2021-02-11T09:50:31.321 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:50:31.317+0000 7fd893c9b080 0 bucket checkpoint complete 2021-02-11T09:50:31.350 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:31.350 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:31.350 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-22 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:31.393 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:31.393 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:31.602 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:50:31.601+0000 7f900cc27080 1 bucket sync caught up with source: 2021-02-11T09:50:31.603 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , 00000000002.16.5, , ] 2021-02-11T09:50:31.603 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , 00000000002.16.5, , ] 2021-02-11T09:50:31.603 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:50:31.601+0000 7f900cc27080 0 bucket checkpoint complete 2021-02-11T09:50:31.631 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:31.631 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:31.631 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-22 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:31.679 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:31.679 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:31.897 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:50:31.891+0000 7fb24bf65080 1 bucket sync caught up with source: 2021-02-11T09:50:31.898 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, , , , , , , , 00000000002.16.5, , ] 2021-02-11T09:50:31.898 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, , , , , , , , 00000000002.16.5, , ] 2021-02-11T09:50:31.898 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:50:31.891+0000 7fb24bf65080 0 bucket checkpoint complete 2021-02-11T09:50:31.931 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-22 zones={test-zone2, test-zone1} 2021-02-11T09:50:31.940 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:50:31.991 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:31.991 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:31.991 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:50:31.998 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:31.998 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:32.057 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:32.066 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:32.077 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:32.144 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-22 zones={test-zone2, test-zone1} 2021-02-11T09:50:32.144 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-22 zones={test-zone3, test-zone1} 2021-02-11T09:50:32.152 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:50:32.202 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:32.203 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:32.203 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:50:32.208 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:32.208 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:32.261 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:32.269 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:32.278 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:32.345 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-22 zones={test-zone3, test-zone1} 2021-02-11T09:50:32.345 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-22 zones={test-zone3, test-zone2} 2021-02-11T09:50:32.357 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:50:32.403 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:32.403 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:32.403 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:50:32.409 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:32.409 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:32.467 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:32.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:32.485 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:32.554 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-22 zones={test-zone3, test-zone2} 2021-02-11T09:50:32.554 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:32.554 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:32.555 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:32.601 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:32.601 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:32.835 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:50:32.831+0000 7fee5477e080 1 bucket sync caught up with source: 2021-02-11T09:50:32.835 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, , , , , , , , 00000000002.28.5, , ] 2021-02-11T09:50:32.835 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, , , , , , , , 00000000002.28.5, , ] 2021-02-11T09:50:32.835 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:50:32.831+0000 7fee5477e080 0 bucket checkpoint complete 2021-02-11T09:50:32.873 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:32.873 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:32.873 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:32.921 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:32.921 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:33.125 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:50:33.125+0000 7f254e7cc080 1 bucket sync caught up with source: 2021-02-11T09:50:33.125 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , 00000000002.28.5, , ] 2021-02-11T09:50:33.125 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , 00000000002.28.5, , ] 2021-02-11T09:50:33.125 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:50:33.125+0000 7f254e7cc080 0 bucket checkpoint complete 2021-02-11T09:50:33.154 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:33.154 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:33.155 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:33.202 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:33.202 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:33.438 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:50:33.431+0000 7f60725d5080 1 bucket sync caught up with source: 2021-02-11T09:50:33.438 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, , , , , , , , 00000000002.28.5, , ] 2021-02-11T09:50:33.438 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, , , , , , , , 00000000002.28.5, , ] 2021-02-11T09:50:33.438 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:50:33.431+0000 7f60725d5080 0 bucket checkpoint complete 2021-02-11T09:50:33.472 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:33.472 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:33.472 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:33.520 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:33.520 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:33.705 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:50:33.700+0000 7f4b95265080 1 bucket sync caught up with source: 2021-02-11T09:50:33.705 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , 00000000002.40.5, , ] 2021-02-11T09:50:33.705 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , 00000000002.40.5, , ] 2021-02-11T09:50:33.705 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:50:33.700+0000 7f4b95265080 0 bucket checkpoint complete 2021-02-11T09:50:33.736 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:33.736 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:33.736 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:33.782 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:33.782 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:33.970 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:50:33.968+0000 7ff6546a6080 1 bucket sync caught up with source: 2021-02-11T09:50:33.970 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , 00000000002.40.5, , ] 2021-02-11T09:50:33.970 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , 00000000002.40.5, , ] 2021-02-11T09:50:33.970 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:50:33.968+0000 7ff6546a6080 0 bucket checkpoint complete 2021-02-11T09:50:34.005 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:34.005 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:34.006 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:34.052 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:34.052 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:34.263 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:50:34.260+0000 7feece155080 1 bucket sync caught up with source: 2021-02-11T09:50:34.263 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , 00000000003.56.11, , ] 2021-02-11T09:50:34.263 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , 00000000003.56.11, , ] 2021-02-11T09:50:34.263 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:50:34.260+0000 7feece155080 0 bucket checkpoint complete 2021-02-11T09:50:34.289 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:34.289 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:34.289 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:34.336 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:34.336 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:34.565 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:50:34.559+0000 7fb0351a5080 1 bucket sync caught up with source: 2021-02-11T09:50:34.565 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, , , , , , , , 00000000003.56.11, , ] 2021-02-11T09:50:34.565 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, , , , , , , , 00000000003.56.11, , ] 2021-02-11T09:50:34.565 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:50:34.559+0000 7fb0351a5080 0 bucket checkpoint complete 2021-02-11T09:50:34.601 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-23 zones={test-zone2, test-zone1} 2021-02-11T09:50:34.611 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:50:34.659 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:34.659 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:34.659 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:50:34.667 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:34.667 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:34.722 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:34.730 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:34.740 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:34.811 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-23 zones={test-zone2, test-zone1} 2021-02-11T09:50:34.811 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-23 zones={test-zone3, test-zone1} 2021-02-11T09:50:34.821 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:50:34.867 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:34.867 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:34.867 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:50:34.873 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:34.873 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:34.929 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:34.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:34.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:35.013 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-23 zones={test-zone3, test-zone1} 2021-02-11T09:50:35.013 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-23 zones={test-zone3, test-zone2} 2021-02-11T09:50:35.023 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:50:35.074 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:35.075 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:35.075 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:50:35.081 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:35.081 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:35.138 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:35.145 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:35.155 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:35.220 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-23 zones={test-zone3, test-zone2} 2021-02-11T09:50:35.220 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_delete_marker_full_sync ... ok 2021-02-11T09:50:35.221 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=tkscyf-24 2021-02-11T09:50:35.253 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=tkscyf-25 2021-02-11T09:50:35.309 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=tkscyf-26 2021-02-11T09:50:35.451 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:50:35.451 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:50:35.451 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:50:35.497 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:35.497 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:35.672 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1 2021-02-11T09:50:35.705 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613037035.445782_288.1","last_update":"2021-02-11T09:50:35.445782Z"},{"marker":"1_1613036864.081584_81.1","last_update":"2021-02-11T09:47:44.081584Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T09:50:35.706 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T09:50:35.706 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:50:35.706 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:50:35.706 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:50:35.747 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:35.747 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:35.988 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037035.445782_288.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:50:35.445782Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:50:35.989 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T09:50:35.989 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T09:50:35.989 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:50:35.989 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:50:35.989 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:50:36.031 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:36.031 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:36.258 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037035.445782_288.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:50:35.445782Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:50:36.258 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T09:50:36.258 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T09:50:36.258 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:50:36.258 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:50:36.259 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:50:36.301 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:36.301 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:36.520 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037035.445782_288.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:50:35.445782Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:50:36.521 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T09:50:36.746 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:36.746 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:36.747 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:36.791 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:36.791 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:36.983 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:50:36.979+0000 7f0815498080 1 bucket sync caught up with source: 2021-02-11T09:50:36.983 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, , , , , , , , 00000000003.21.6, , ] 2021-02-11T09:50:36.984 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, , , , , , , , 00000000003.21.6, , ] 2021-02-11T09:50:36.984 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:50:36.979+0000 7f0815498080 0 bucket checkpoint complete 2021-02-11T09:50:37.013 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:37.013 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:37.013 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:37.056 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:37.056 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:37.256 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:50:37.252+0000 7f9c8ef09080 1 bucket sync caught up with source: 2021-02-11T09:50:37.256 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , 00000000003.21.6, , ] 2021-02-11T09:50:37.256 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , 00000000003.21.6, , ] 2021-02-11T09:50:37.256 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:50:37.252+0000 7f9c8ef09080 0 bucket checkpoint complete 2021-02-11T09:50:37.285 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:37.285 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:37.285 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:37.338 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:37.338 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:37.541 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:50:37.535+0000 7f2a8bf67080 1 bucket sync caught up with source: 2021-02-11T09:50:37.541 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, , , , , , , , 00000000003.21.6, , ] 2021-02-11T09:50:37.541 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, , , , , , , , 00000000003.21.6, , ] 2021-02-11T09:50:37.542 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:50:37.535+0000 7f2a8bf67080 0 bucket checkpoint complete 2021-02-11T09:50:37.574 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:37.574 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:37.574 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-24 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:37.623 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:37.623 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:37.836 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:50:37.832+0000 7f5fb5c2c080 1 bucket sync caught up with source: 2021-02-11T09:50:37.836 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , 00000000001.24.12, , ] 2021-02-11T09:50:37.836 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , 00000000001.24.12, , ] 2021-02-11T09:50:37.836 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:50:37.832+0000 7f5fb5c2c080 0 bucket checkpoint complete 2021-02-11T09:50:37.870 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:37.870 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:37.870 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-24 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:37.917 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:37.917 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:38.125 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:50:38.124+0000 7f3d39998080 1 bucket sync caught up with source: 2021-02-11T09:50:38.126 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , 00000000001.24.12, , ] 2021-02-11T09:50:38.126 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , 00000000001.24.12, , ] 2021-02-11T09:50:38.126 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:50:38.124+0000 7f3d39998080 0 bucket checkpoint complete 2021-02-11T09:50:38.159 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:38.159 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:38.160 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-24 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:38.206 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:38.206 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:38.410 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:50:38.408+0000 7fe8ae396080 1 waiting for incremental sync to catch up: 2021-02-11T09:50:38.410 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , , , ] 2021-02-11T09:50:38.410 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , 00000000001.22.12, , ] 2021-02-11T09:50:43.413 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:50:43.408+0000 7fe8ae396080 1 bucket sync caught up with source: 2021-02-11T09:50:43.414 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , 00000000001.22.12, , ] 2021-02-11T09:50:43.414 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , 00000000001.22.12, , ] 2021-02-11T09:50:43.414 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:50:43.408+0000 7fe8ae396080 0 bucket checkpoint complete 2021-02-11T09:50:43.444 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:43.444 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:43.444 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-24 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:43.492 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:43.492 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:43.706 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:50:43.703+0000 7fe5cdb6e080 1 bucket sync caught up with source: 2021-02-11T09:50:43.706 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, , , , , , , , 00000000001.22.12, , ] 2021-02-11T09:50:43.707 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, , , , , , , , 00000000001.22.12, , ] 2021-02-11T09:50:43.707 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:50:43.703+0000 7fe5cdb6e080 0 bucket checkpoint complete 2021-02-11T09:50:43.739 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-24 zones={test-zone2, test-zone1} 2021-02-11T09:50:43.749 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:50:43.799 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:43.799 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:50:43.806 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:43.860 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:43.866 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-24 zones={test-zone2, test-zone1} 2021-02-11T09:50:43.866 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-24 zones={test-zone3, test-zone1} 2021-02-11T09:50:43.874 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:50:43.922 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:43.923 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:50:43.929 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:43.985 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:43.990 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-24 zones={test-zone3, test-zone1} 2021-02-11T09:50:43.990 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-24 zones={test-zone3, test-zone2} 2021-02-11T09:50:43.999 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:50:44.050 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:44.050 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:50:44.057 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:44.113 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:44.118 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-24 zones={test-zone3, test-zone2} 2021-02-11T09:50:44.119 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:44.119 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:44.119 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-25 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:44.162 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:44.163 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:44.358 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:50:44.355+0000 7fcb097c8080 1 bucket sync caught up with source: 2021-02-11T09:50:44.358 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, , , , , , , , 00000000001.8.12, , ] 2021-02-11T09:50:44.358 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, , , , , , , , 00000000001.8.12, , ] 2021-02-11T09:50:44.358 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:50:44.355+0000 7fcb097c8080 0 bucket checkpoint complete 2021-02-11T09:50:44.385 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:44.385 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:44.386 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-25 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:44.435 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:44.435 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:44.643 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:50:44.640+0000 7f27c691c080 1 bucket sync caught up with source: 2021-02-11T09:50:44.643 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , 00000000001.8.12, , ] 2021-02-11T09:50:44.644 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , 00000000001.8.12, , ] 2021-02-11T09:50:44.644 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:50:44.640+0000 7f27c691c080 0 bucket checkpoint complete 2021-02-11T09:50:44.672 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:44.672 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:44.672 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-25 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:44.719 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:44.720 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:44.946 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:50:44.939+0000 7fc43e983080 1 bucket sync caught up with source: 2021-02-11T09:50:44.946 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, , , , , , , , 00000000001.8.12, , ] 2021-02-11T09:50:44.947 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, , , , , , , , 00000000001.8.12, , ] 2021-02-11T09:50:44.947 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:50:44.939+0000 7fc43e983080 0 bucket checkpoint complete 2021-02-11T09:50:44.977 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:44.978 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:44.978 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-25 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:45.025 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:45.026 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:45.241 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:50:45.236+0000 7f760bc44080 1 bucket sync caught up with source: 2021-02-11T09:50:45.241 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , 00000000003.23.6, , ] 2021-02-11T09:50:45.241 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , 00000000003.23.6, , ] 2021-02-11T09:50:45.241 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:50:45.236+0000 7f760bc44080 0 bucket checkpoint complete 2021-02-11T09:50:45.277 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:45.277 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:45.277 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-25 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:45.325 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:45.325 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:45.519 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:50:45.516+0000 7fd47859d080 1 bucket sync caught up with source: 2021-02-11T09:50:45.519 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , 00000000003.23.6, , ] 2021-02-11T09:50:45.519 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , 00000000003.23.6, , ] 2021-02-11T09:50:45.519 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:50:45.516+0000 7fd47859d080 0 bucket checkpoint complete 2021-02-11T09:50:45.552 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:45.552 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:45.553 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-25 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:45.641 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:45.641 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:45.853 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:50:45.848+0000 7fe8aafbe080 1 bucket sync caught up with source: 2021-02-11T09:50:45.853 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , 00000000001.14.12, , ] 2021-02-11T09:50:45.853 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , 00000000001.14.12, , ] 2021-02-11T09:50:45.854 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:50:45.848+0000 7fe8aafbe080 0 bucket checkpoint complete 2021-02-11T09:50:45.883 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:45.883 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:45.883 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-25 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:45.931 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:45.931 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:46.150 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:50:46.143+0000 7ffbc21bc080 1 bucket sync caught up with source: 2021-02-11T09:50:46.150 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, , , , , , , , 00000000001.14.12, , ] 2021-02-11T09:50:46.150 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, , , , , , , , 00000000001.14.12, , ] 2021-02-11T09:50:46.150 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:50:46.143+0000 7ffbc21bc080 0 bucket checkpoint complete 2021-02-11T09:50:46.185 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-25 zones={test-zone2, test-zone1} 2021-02-11T09:50:46.198 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:50:46.249 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:46.249 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:50:46.258 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:46.326 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:46.337 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-25 zones={test-zone2, test-zone1} 2021-02-11T09:50:46.337 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-25 zones={test-zone3, test-zone1} 2021-02-11T09:50:46.349 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:50:46.395 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:46.395 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:50:46.400 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:46.452 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:46.457 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-25 zones={test-zone3, test-zone1} 2021-02-11T09:50:46.457 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-25 zones={test-zone3, test-zone2} 2021-02-11T09:50:46.467 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:50:46.514 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:46.515 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:50:46.522 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:46.582 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:46.587 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-25 zones={test-zone3, test-zone2} 2021-02-11T09:50:46.588 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:46.588 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:46.588 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-26 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:46.630 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:46.630 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:46.861 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:50:46.855+0000 7fd09c46a080 1 bucket sync caught up with source: 2021-02-11T09:50:46.862 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, , , , , , , , 00000000001.35.12, , ] 2021-02-11T09:50:46.862 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, , , , , , , , 00000000001.35.12, , ] 2021-02-11T09:50:46.862 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:50:46.855+0000 7fd09c46a080 0 bucket checkpoint complete 2021-02-11T09:50:46.895 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:46.896 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:46.896 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-26 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:46.942 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:46.942 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:47.153 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:50:47.152+0000 7f40d6cf2080 1 bucket sync caught up with source: 2021-02-11T09:50:47.154 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , 00000000001.35.12, , ] 2021-02-11T09:50:47.154 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , 00000000001.35.12, , ] 2021-02-11T09:50:47.154 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:50:47.152+0000 7f40d6cf2080 0 bucket checkpoint complete 2021-02-11T09:50:47.180 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:47.180 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:47.180 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-26 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:47.227 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:47.227 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:47.427 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:50:47.423+0000 7fedfe939080 1 bucket sync caught up with source: 2021-02-11T09:50:47.427 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, , , , , , , , 00000000001.35.12, , ] 2021-02-11T09:50:47.427 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, , , , , , , , 00000000001.35.12, , ] 2021-02-11T09:50:47.427 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:50:47.423+0000 7fedfe939080 0 bucket checkpoint complete 2021-02-11T09:50:47.457 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:47.457 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:47.457 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-26 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:47.505 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:47.505 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:47.711 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:50:47.708+0000 7f99171b1080 1 bucket sync caught up with source: 2021-02-11T09:50:47.712 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , 00000000001.21.12, , ] 2021-02-11T09:50:47.712 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , 00000000001.21.12, , ] 2021-02-11T09:50:47.712 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:50:47.708+0000 7f99171b1080 0 bucket checkpoint complete 2021-02-11T09:50:47.745 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:47.746 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:47.746 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-26 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:47.794 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:47.794 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:48.001 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:50:47.996+0000 7fbb63958080 1 bucket sync caught up with source: 2021-02-11T09:50:48.001 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , 00000000001.21.12, , ] 2021-02-11T09:50:48.002 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , 00000000001.21.12, , ] 2021-02-11T09:50:48.002 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:50:47.996+0000 7fbb63958080 0 bucket checkpoint complete 2021-02-11T09:50:48.034 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:48.034 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:48.034 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-26 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:48.082 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:48.082 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:48.295 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:50:48.292+0000 7f17a3fe5080 1 bucket sync caught up with source: 2021-02-11T09:50:48.295 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , 00000000003.30.6, , ] 2021-02-11T09:50:48.295 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , 00000000003.30.6, , ] 2021-02-11T09:50:48.295 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:50:48.292+0000 7f17a3fe5080 0 bucket checkpoint complete 2021-02-11T09:50:48.324 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:48.325 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T09:50:48.325 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-26 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T09:50:48.379 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:48.379 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:48.587 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:50:48.583+0000 7f2e19319080 1 bucket sync caught up with source: 2021-02-11T09:50:48.587 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, , , , , , , , 00000000003.30.6, , ] 2021-02-11T09:50:48.587 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, , , , , , , , 00000000003.30.6, , ] 2021-02-11T09:50:48.587 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:50:48.583+0000 7f2e19319080 0 bucket checkpoint complete 2021-02-11T09:50:48.621 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-26 zones={test-zone2, test-zone1} 2021-02-11T09:50:48.632 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:50:48.679 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:48.679 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:50:48.686 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:48.742 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:48.747 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-26 zones={test-zone2, test-zone1} 2021-02-11T09:50:48.748 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-26 zones={test-zone3, test-zone1} 2021-02-11T09:50:48.757 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:50:48.803 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:48.803 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:50:48.809 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:48.864 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:48.870 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-26 zones={test-zone3, test-zone1} 2021-02-11T09:50:48.870 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-26 zones={test-zone3, test-zone2} 2021-02-11T09:50:48.880 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:50:48.926 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:48.927 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:50:48.934 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T09:50:48.989 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T09:50:48.994 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-26 zones={test-zone3, test-zone2} 2021-02-11T09:50:48.994 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_suspended_delete_marker_full_sync ... ok 2021-02-11T09:50:48.995 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=tkscyf-27 2021-02-11T09:50:49.030 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=tkscyf-28 2021-02-11T09:50:49.110 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=tkscyf-29 2021-02-11T09:50:49.350 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_versioning ... ok 2021-02-11T09:50:49.351 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=tkscyf-30 2021-02-11T09:50:49.381 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=tkscyf-31 2021-02-11T09:50:49.437 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=tkscyf-32 2021-02-11T09:50:49.658 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_acl ... ok 2021-02-11T09:50:49.659 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=tkscyf-33 2021-02-11T09:50:49.702 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=tkscyf-34 2021-02-11T09:50:49.763 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=tkscyf-35 2021-02-11T09:50:51.999 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_cors ... ok 2021-02-11T09:50:56.376 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=tkscyf-36 2021-02-11T09:50:56.412 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=tkscyf-37 2021-02-11T09:50:56.475 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=tkscyf-38 2021-02-11T09:50:56.536 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:50:56.537 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:50:56.537 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:50:56.568 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:56.568 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:56.743 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1 2021-02-11T09:50:56.768 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613037056.514482_354.1","last_update":"2021-02-11T09:50:56.514482Z"},{"marker":"1_1613036864.081584_81.1","last_update":"2021-02-11T09:47:44.081584Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T09:50:56.768 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T09:50:56.769 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:50:56.769 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:50:56.769 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:50:56.806 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:56.807 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:57.035 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037056.514482_354.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:50:56.514482Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:50:57.036 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T09:50:57.036 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T09:50:57.036 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:50:57.036 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:50:57.036 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:50:57.078 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:57.078 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:57.291 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037056.514482_354.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:50:56.514482Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:50:57.292 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T09:50:57.292 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T09:50:57.292 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:50:57.292 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:50:57.292 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:50:57.333 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:57.333 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:57.551 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037056.514482_354.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:50:56.514482Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:50:57.552 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T09:50:57.896 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_delete_notempty ... ok 2021-02-11T09:50:57.896 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=tkscyf-39 2021-02-11T09:50:57.927 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=tkscyf-40 2021-02-11T09:50:57.980 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=tkscyf-41 2021-02-11T09:50:58.024 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:50:58.024 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:50:58.025 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:50:58.056 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:58.056 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:58.209 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1 2021-02-11T09:50:58.243 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613037058.005761_366.1","last_update":"2021-02-11T09:50:58.005761Z"},{"marker":"1_1613036864.081584_81.1","last_update":"2021-02-11T09:47:44.081584Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T09:50:58.244 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T09:50:58.244 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:50:58.244 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:50:58.244 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:50:58.285 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:58.285 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:58.496 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037058.005761_366.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:50:58.005761Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:50:58.497 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T09:50:58.497 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T09:50:58.497 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:50:58.497 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:50:58.497 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:50:58.540 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:58.540 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:58.765 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037058.005761_366.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:50:58.005761Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:50:58.765 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T09:50:58.766 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T09:50:58.766 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:50:58.766 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:50:58.766 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:50:58.807 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:50:58.807 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:50:59.041 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037058.005761_366.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:50:58.005761Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:50:59.041 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T09:50:59.041 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2021-02-11T09:50:59.041 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T09:50:59.042 INFO:tasks.rgw.c1.client.1.smithi094.stdout:2021-02-11T09:50:59.040+0000 7fd3eeffd700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 19694) UID: 0 2021-02-11T09:50:59.042 INFO:tasks.rgw.c1.client.1.smithi094.stdout:2021-02-11T09:50:59.040+0000 7fd40a8c00c0 -1 shutting down 2021-02-11T09:51:02.690 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2021-02-11T09:51:05.143 INFO:tasks.rgw.c1.client.1:Stopped 2021-02-11T09:51:05.143 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:51:05.143 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:51:05.143 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --master --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T09:51:05.184 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:51:05.185 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:51:05.211 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:51:05.210+0000 7f044dddd080 0 NOTICE: overriding master zone: dd308248-8ee8-49be-b928-07c3de7d7aa2 2021-02-11T09:51:05.214 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:51:05.210+0000 7f044dddd080 0 failed reading obj info from .rgw.root:zone_info.92d097d8-38ae-4839-9777-3b30db2b2f60: (2) No such file or directory 2021-02-11T09:51:05.214 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:51:05.210+0000 7f044dddd080 0 WARNING: could not read zone params for zone id=92d097d8-38ae-4839-9777-3b30db2b2f60 name=test-zone3 2021-02-11T09:51:05.215 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:51:05.214+0000 7f044dddd080 0 failed reading obj info from .rgw.root:zone_info.dd308248-8ee8-49be-b928-07c3de7d7aa2: (2) No such file or directory 2021-02-11T09:51:05.215 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:51:05.214+0000 7f044dddd080 0 WARNING: could not read zone params for zone id=dd308248-8ee8-49be-b928-07c3de7d7aa2 name=test-zone1 2021-02-11T09:51:05.227 INFO:teuthology.orchestra.run.smithi098.stdout:{"id":"63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"7B4L28ZSM4TSLFS6","secret_key":"eZwNt56BcNJm7h6ZlwjNkhL3ytjGBlGP"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"7e807acb-79b4-4a09-ba75-06bbedad4839","notif_pool":"test-zone2.rgw.log:notif"} 2021-02-11T09:51:05.228 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:51:05.228 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:51:05.228 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T09:51:05.307 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:51:05.307 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:51:05.617 INFO:teuthology.orchestra.run.smithi098.stdout:{"id":"47caa906-66a7-4be6-bbb9-9f539accb732","epoch":1,"predecessor_uuid":"ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1","sync_status":["1_1613037058.005761_366.1","1_1613036864.081584_81.1","",""],"period_map":{"id":"47caa906-66a7-4be6-bbb9-9f539accb732","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi094.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2","zones":[{"id":"45265506-b5d1-4710-9a9e-3e67bee0e451","name":"test-zone4","endpoints":["http://smithi098.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2","name":"test-zone2","endpoints":["http://smithi098.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"92d097d8-38ae-4839-9777-3b30db2b2f60","name":"test-zone3","endpoints":["http://smithi094.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"dd308248-8ee8-49be-b928-07c3de7d7aa2","name":"test-zone1","endpoints":["http://smithi094.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"7e807acb-79b4-4a09-ba75-06bbedad4839","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"45265506-b5d1-4710-9a9e-3e67bee0e451","val":240795374},{"key":"63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2","val":1552258706},{"key":"92d097d8-38ae-4839-9777-3b30db2b2f60","val":2008974102},{"key":"dd308248-8ee8-49be-b928-07c3de7d7aa2","val":207523491}]},"master_zonegroup":"test-zonegroup","master_zone":"63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"7e807acb-79b4-4a09-ba75-06bbedad4839","realm_name":"test-realm","realm_epoch":3} 2021-02-11T09:51:05.618 INFO:rgw_multi.tests:Set master zone=test-zone2, waiting 60s for reconfiguration.. 2021-02-11T09:51:08.296 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2021-02-11T09:52:05.619 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=tkscyf-42 2021-02-11T09:52:05.694 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=tkscyf-43 2021-02-11T09:52:05.731 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:52:05.732 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:52:05.732 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:52:05.767 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:52:05.767 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:52:05.973 INFO:teuthology.orchestra.run.smithi098.stderr:No --period given, using current period=47caa906-66a7-4be6-bbb9-9f539accb732 2021-02-11T09:52:06.008 INFO:teuthology.orchestra.run.smithi098.stdout:[{"marker":"1_1613037125.727813_116.1","last_update":"2021-02-11T09:52:05.727813Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T09:52:06.009 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone1 2021-02-11T09:52:06.009 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:52:06.009 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:52:06.009 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:52:06.051 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:52:06.052 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:52:06.284 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"47caa906-66a7-4be6-bbb9-9f539accb732","realm_epoch":3},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037125.727813_116.1","next_step_marker":"","total_entries":80,"pos":0,"timestamp":"2021-02-11T09:52:05.727813Z","realm_epoch":3}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":3}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":3}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":3}}]},"full_sync":{"total":80,"complete":80}} 2021-02-11T09:52:06.285 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone1 2021-02-11T09:52:06.285 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:52:06.285 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:52:06.285 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --master --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T09:52:06.367 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:52:06.367 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:52:06.391 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:52:06.386+0000 7f904c46b080 0 NOTICE: overriding master zone: 63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2 2021-02-11T09:52:06.391 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:52:06.386+0000 7f904c46b080 0 failed reading obj info from .rgw.root:zone_info.45265506-b5d1-4710-9a9e-3e67bee0e451: (2) No such file or directory 2021-02-11T09:52:06.391 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:52:06.386+0000 7f904c46b080 0 WARNING: could not read zone params for zone id=45265506-b5d1-4710-9a9e-3e67bee0e451 name=test-zone4 2021-02-11T09:52:06.392 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:52:06.386+0000 7f904c46b080 0 failed reading obj info from .rgw.root:zone_info.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2: (2) No such file or directory 2021-02-11T09:52:06.392 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:52:06.386+0000 7f904c46b080 0 WARNING: could not read zone params for zone id=63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2 name=test-zone2 2021-02-11T09:52:06.406 INFO:teuthology.orchestra.run.smithi094.stdout:{"id":"dd308248-8ee8-49be-b928-07c3de7d7aa2","name":"test-zone1","domain_root":"test-zone1.rgw.meta:root","control_pool":"test-zone1.rgw.control","gc_pool":"test-zone1.rgw.log:gc","lc_pool":"test-zone1.rgw.log:lc","log_pool":"test-zone1.rgw.log","intent_log_pool":"test-zone1.rgw.log:intent","usage_log_pool":"test-zone1.rgw.log:usage","roles_pool":"test-zone1.rgw.meta:roles","reshard_pool":"test-zone1.rgw.log:reshard","user_keys_pool":"test-zone1.rgw.meta:users.keys","user_email_pool":"test-zone1.rgw.meta:users.email","user_swift_pool":"test-zone1.rgw.meta:users.swift","user_uid_pool":"test-zone1.rgw.meta:users.uid","otp_pool":"test-zone1.rgw.otp","system_key":{"access_key":"7B4L28ZSM4TSLFS6","secret_key":"eZwNt56BcNJm7h6ZlwjNkhL3ytjGBlGP"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone1.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone1.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone1.rgw.buckets.non-ec","index_type":0}}],"realm_id":"7e807acb-79b4-4a09-ba75-06bbedad4839","notif_pool":"test-zone1.rgw.log:notif"} 2021-02-11T09:52:06.406 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:52:06.406 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:52:06.407 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T09:52:06.490 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:52:06.490 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:52:06.798 INFO:teuthology.orchestra.run.smithi094.stdout:{"id":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","epoch":1,"predecessor_uuid":"47caa906-66a7-4be6-bbb9-9f539accb732","sync_status":["1_1613037125.727813_116.1","","",""],"period_map":{"id":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi094.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"dd308248-8ee8-49be-b928-07c3de7d7aa2","zones":[{"id":"45265506-b5d1-4710-9a9e-3e67bee0e451","name":"test-zone4","endpoints":["http://smithi098.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2","name":"test-zone2","endpoints":["http://smithi098.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"92d097d8-38ae-4839-9777-3b30db2b2f60","name":"test-zone3","endpoints":["http://smithi094.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"dd308248-8ee8-49be-b928-07c3de7d7aa2","name":"test-zone1","endpoints":["http://smithi094.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"7e807acb-79b4-4a09-ba75-06bbedad4839","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"45265506-b5d1-4710-9a9e-3e67bee0e451","val":240795374},{"key":"63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2","val":1552258706},{"key":"92d097d8-38ae-4839-9777-3b30db2b2f60","val":2008974102},{"key":"dd308248-8ee8-49be-b928-07c3de7d7aa2","val":207523491}]},"master_zonegroup":"test-zonegroup","master_zone":"dd308248-8ee8-49be-b928-07c3de7d7aa2","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"7e807acb-79b4-4a09-ba75-06bbedad4839","realm_name":"test-realm","realm_epoch":4} 2021-02-11T09:52:06.798 INFO:rgw_multi.tests:Set master zone=test-zone1, waiting 60s for reconfiguration.. 2021-02-11T09:53:06.799 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=tkscyf-44 2021-02-11T09:53:06.845 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=tkscyf-45 2021-02-11T09:53:06.915 INFO:tasks.rgw.c1.client.1:Restarting daemon 2021-02-11T09:53:06.915 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2021-02-11T09:53:06.918 INFO:tasks.rgw.c1.client.1:Started 2021-02-11T09:53:06.918 DEBUG:teuthology.orchestra.run.smithi094:> curl http://smithi094.front.sepia.ceph.com:8001 2021-02-11T09:53:06.933 INFO:teuthology.orchestra.run.smithi094.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-11T09:53:06.933 INFO:teuthology.orchestra.run.smithi094.stderr: Dload Upload Total Spent Left Speed 2021-02-11T09:53:06.937 INFO:teuthology.orchestra.run.smithi094.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi094.front.sepia.ceph.com port 8001: Connection refused 2021-02-11T09:53:06.938 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-02-11T09:53:07.939 DEBUG:teuthology.orchestra.run.smithi094:> curl http://smithi094.front.sepia.ceph.com:8001 2021-02-11T09:53:07.953 INFO:teuthology.orchestra.run.smithi094.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-11T09:53:07.953 INFO:teuthology.orchestra.run.smithi094.stderr: Dload Upload Total Spent Left Speed 2021-02-11T09:53:07.961 INFO:teuthology.orchestra.run.smithi094.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 30571 0 --:--:-- --:--:-- --:--:-- 30571 2021-02-11T09:53:07.961 INFO:teuthology.orchestra.run.smithi094.stdout:anonymous 2021-02-11T09:53:07.961 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:07.962 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:07.962 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:08.040 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:08.040 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:08.222 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 2021-02-11T09:53:08.253 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613037186.890832_249.1","last_update":"2021-02-11T09:53:06.890832Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T09:53:08.253 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T09:53:08.254 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:08.254 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:08.254 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:08.299 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:08.299 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:08.552 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037186.890832_249.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:53:06.890832Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:53:08.552 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T09:53:08.552 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T09:53:08.552 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:08.552 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:08.553 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:08.593 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:08.593 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:08.822 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037186.890832_249.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:53:06.890832Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:53:08.822 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T09:53:08.822 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T09:53:08.823 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:08.823 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:08.823 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:08.865 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:08.865 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:09.114 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037186.890832_249.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:53:06.890832Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:53:09.115 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T09:53:09.115 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-39 zones={test-zone2, test-zone1} 2021-02-11T09:53:09.135 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:53:09.183 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:53:09.246 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-39 zones={test-zone2, test-zone1} 2021-02-11T09:53:09.246 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-39 zones={test-zone3, test-zone1} 2021-02-11T09:53:09.262 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:53:09.311 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:53:09.373 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-39 zones={test-zone3, test-zone1} 2021-02-11T09:53:09.373 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-39 zones={test-zone1, test-zone2} 2021-02-11T09:53:09.381 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:53:09.427 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:53:09.489 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-39 zones={test-zone1, test-zone2} 2021-02-11T09:53:09.489 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-39 zones={test-zone3, test-zone2} 2021-02-11T09:53:09.499 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:53:09.546 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:53:09.604 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-39 zones={test-zone3, test-zone2} 2021-02-11T09:53:09.605 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-39 zones={test-zone1, test-zone3} 2021-02-11T09:53:09.616 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:53:09.662 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:53:09.721 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-39 zones={test-zone1, test-zone3} 2021-02-11T09:53:09.721 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-39 zones={test-zone2, test-zone3} 2021-02-11T09:53:09.730 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:53:09.778 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:53:09.837 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-39 zones={test-zone2, test-zone3} 2021-02-11T09:53:09.837 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-40 zones={test-zone2, test-zone1} 2021-02-11T09:53:09.851 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:53:09.899 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:53:09.970 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-40 zones={test-zone2, test-zone1} 2021-02-11T09:53:09.970 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-40 zones={test-zone3, test-zone1} 2021-02-11T09:53:09.981 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:53:10.026 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:53:10.089 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-40 zones={test-zone3, test-zone1} 2021-02-11T09:53:10.089 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-40 zones={test-zone1, test-zone2} 2021-02-11T09:53:10.098 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:53:10.151 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:53:10.213 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-40 zones={test-zone1, test-zone2} 2021-02-11T09:53:10.213 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-40 zones={test-zone3, test-zone2} 2021-02-11T09:53:10.223 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:53:10.270 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:53:10.329 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-40 zones={test-zone3, test-zone2} 2021-02-11T09:53:10.330 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-40 zones={test-zone1, test-zone3} 2021-02-11T09:53:10.343 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:53:10.395 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:53:10.456 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-40 zones={test-zone1, test-zone3} 2021-02-11T09:53:10.456 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-40 zones={test-zone2, test-zone3} 2021-02-11T09:53:10.466 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:53:10.515 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:53:10.581 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-40 zones={test-zone2, test-zone3} 2021-02-11T09:53:10.581 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-41 zones={test-zone2, test-zone1} 2021-02-11T09:53:10.596 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:53:10.646 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:53:10.714 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-41 zones={test-zone2, test-zone1} 2021-02-11T09:53:10.714 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-41 zones={test-zone3, test-zone1} 2021-02-11T09:53:10.725 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:53:10.775 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:53:10.838 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-41 zones={test-zone3, test-zone1} 2021-02-11T09:53:10.838 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-41 zones={test-zone1, test-zone2} 2021-02-11T09:53:10.849 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:53:10.895 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:53:10.957 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-41 zones={test-zone1, test-zone2} 2021-02-11T09:53:10.958 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-41 zones={test-zone3, test-zone2} 2021-02-11T09:53:10.968 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:53:11.015 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:53:11.077 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-41 zones={test-zone3, test-zone2} 2021-02-11T09:53:11.078 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-41 zones={test-zone1, test-zone3} 2021-02-11T09:53:11.088 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:53:11.135 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:53:11.193 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-41 zones={test-zone1, test-zone3} 2021-02-11T09:53:11.194 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-41 zones={test-zone2, test-zone3} 2021-02-11T09:53:11.204 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:53:11.251 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:53:11.311 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-41 zones={test-zone2, test-zone3} 2021-02-11T09:53:11.312 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-42 zones={test-zone2, test-zone1} 2021-02-11T09:53:11.320 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:53:11.367 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:53:11.430 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-42 zones={test-zone2, test-zone1} 2021-02-11T09:53:11.430 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-42 zones={test-zone3, test-zone1} 2021-02-11T09:53:11.440 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:53:11.487 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:53:11.545 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-42 zones={test-zone3, test-zone1} 2021-02-11T09:53:11.545 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-42 zones={test-zone1, test-zone2} 2021-02-11T09:53:11.557 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:53:11.603 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:53:11.665 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-42 zones={test-zone1, test-zone2} 2021-02-11T09:53:11.665 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-42 zones={test-zone3, test-zone2} 2021-02-11T09:53:11.674 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:53:11.726 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:53:11.789 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-42 zones={test-zone3, test-zone2} 2021-02-11T09:53:11.789 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-42 zones={test-zone1, test-zone3} 2021-02-11T09:53:11.800 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:53:11.847 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:53:11.904 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-42 zones={test-zone1, test-zone3} 2021-02-11T09:53:11.904 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-42 zones={test-zone2, test-zone3} 2021-02-11T09:53:11.914 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:53:11.962 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:53:12.029 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-42 zones={test-zone2, test-zone3} 2021-02-11T09:53:12.029 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-43 zones={test-zone2, test-zone1} 2021-02-11T09:53:12.040 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:53:12.087 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:53:12.149 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-43 zones={test-zone2, test-zone1} 2021-02-11T09:53:12.150 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-43 zones={test-zone3, test-zone1} 2021-02-11T09:53:12.159 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:53:12.207 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:53:12.268 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-43 zones={test-zone3, test-zone1} 2021-02-11T09:53:12.269 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-43 zones={test-zone1, test-zone2} 2021-02-11T09:53:12.280 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:53:12.326 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:53:12.388 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-43 zones={test-zone1, test-zone2} 2021-02-11T09:53:12.388 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-43 zones={test-zone3, test-zone2} 2021-02-11T09:53:12.396 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:53:12.446 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:53:12.509 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-43 zones={test-zone3, test-zone2} 2021-02-11T09:53:12.509 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-43 zones={test-zone1, test-zone3} 2021-02-11T09:53:12.521 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:53:12.566 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:53:12.628 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-43 zones={test-zone1, test-zone3} 2021-02-11T09:53:12.629 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-43 zones={test-zone2, test-zone3} 2021-02-11T09:53:12.639 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:53:12.686 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:53:12.745 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-43 zones={test-zone2, test-zone3} 2021-02-11T09:53:12.745 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-44 zones={test-zone2, test-zone1} 2021-02-11T09:53:12.755 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:53:12.807 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:53:12.869 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-44 zones={test-zone2, test-zone1} 2021-02-11T09:53:12.869 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-44 zones={test-zone3, test-zone1} 2021-02-11T09:53:12.879 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:53:12.926 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:53:12.989 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-44 zones={test-zone3, test-zone1} 2021-02-11T09:53:12.989 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-44 zones={test-zone1, test-zone2} 2021-02-11T09:53:13.000 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:53:13.046 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:53:13.109 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-44 zones={test-zone1, test-zone2} 2021-02-11T09:53:13.109 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-44 zones={test-zone3, test-zone2} 2021-02-11T09:53:13.119 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:53:13.170 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:53:13.229 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-44 zones={test-zone3, test-zone2} 2021-02-11T09:53:13.229 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-44 zones={test-zone1, test-zone3} 2021-02-11T09:53:13.241 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:53:13.290 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:53:13.352 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-44 zones={test-zone1, test-zone3} 2021-02-11T09:53:13.353 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-44 zones={test-zone2, test-zone3} 2021-02-11T09:53:13.361 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:53:13.410 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:53:13.472 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-44 zones={test-zone2, test-zone3} 2021-02-11T09:53:13.472 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-45 zones={test-zone2, test-zone1} 2021-02-11T09:53:13.483 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:53:13.534 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:53:13.597 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-45 zones={test-zone2, test-zone1} 2021-02-11T09:53:13.597 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-45 zones={test-zone3, test-zone1} 2021-02-11T09:53:13.607 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:53:13.654 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:53:13.712 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-45 zones={test-zone3, test-zone1} 2021-02-11T09:53:13.713 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-45 zones={test-zone1, test-zone2} 2021-02-11T09:53:13.724 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:53:13.770 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:53:13.837 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-45 zones={test-zone1, test-zone2} 2021-02-11T09:53:13.837 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-45 zones={test-zone3, test-zone2} 2021-02-11T09:53:13.847 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:53:13.895 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:53:13.958 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-45 zones={test-zone3, test-zone2} 2021-02-11T09:53:13.958 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-45 zones={test-zone1, test-zone3} 2021-02-11T09:53:13.969 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:53:14.019 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:53:14.085 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-45 zones={test-zone1, test-zone3} 2021-02-11T09:53:14.085 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-45 zones={test-zone2, test-zone3} 2021-02-11T09:53:14.095 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T09:53:14.146 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T09:53:14.209 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-45 zones={test-zone2, test-zone3} 2021-02-11T09:53:14.210 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:14.210 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:14.210 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:14.244 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:14.244 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:14.480 INFO:teuthology.orchestra.run.smithi094.stdout:[{"id":"1_1613036708.088554_48.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2021-02-11T09:45:08.088554566Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RA3OvQQVI2OcioTs6vG61ts","ver":1},"status":{"status":"write"}}},{"id":"1_1613036708.096437_52.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2021-02-11T09:45:08.096437057Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RA3OvQQVI2OcioTs6vG61ts","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036865.186535_60.1","section":"bucket.instance","name":"tkscyf-10:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.10","timestamp":"2021-02-11T09:47:45.186535085Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BOQUev-Msz6Gg8HQx3I-Uw_","ver":1},"status":{"status":"write"}}},{"id":"1_1613036865.191608_62.1","section":"bucket.instance","name":"tkscyf-10:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.10","timestamp":"2021-02-11T09:47:45.191608247Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BOQUev-Msz6Gg8HQx3I-Uw_","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036865.193557_63.1","section":"bucket","name":"tkscyf-10","timestamp":"2021-02-11T09:47:45.193557933Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fhFnaBfHBt4OCc3A0Cy3U0l","ver":1},"status":{"status":"write"}}},{"id":"1_1613036865.198089_64.1","section":"bucket","name":"tkscyf-10","timestamp":"2021-02-11T09:47:45.198089937Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fhFnaBfHBt4OCc3A0Cy3U0l","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036865.217116_66.1","section":"bucket.instance","name":"tkscyf-11:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.11","timestamp":"2021-02-11T09:47:45.217116956Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_x_UhJIyltXoCF5sGbvGrvWs","ver":1},"status":{"status":"write"}}},{"id":"1_1613036865.222221_67.1","section":"bucket.instance","name":"tkscyf-11:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.11","timestamp":"2021-02-11T09:47:45.222221108Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_x_UhJIyltXoCF5sGbvGrvWs","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036865.224285_68.1","section":"bucket","name":"tkscyf-11","timestamp":"2021-02-11T09:47:45.224285083Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FRR9uzZkCcSEP86_Kp419wb","ver":1},"status":{"status":"write"}}},{"id":"1_1613036865.228963_69.1","section":"bucket","name":"tkscyf-11","timestamp":"2021-02-11T09:47:45.228963299Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FRR9uzZkCcSEP86_Kp419wb","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036865.267357_70.1","section":"bucket.instance","name":"tkscyf-12:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.12","timestamp":"2021-02-11T09:47:45.267357171Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QtLjmo_4M7YVwSusOVuvKW7","ver":1},"status":{"status":"write"}}},{"id":"1_1613036865.272260_71.1","section":"bucket.instance","name":"tkscyf-12:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.12","timestamp":"2021-02-11T09:47:45.272260535Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QtLjmo_4M7YVwSusOVuvKW7","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036865.273997_72.1","section":"bucket","name":"tkscyf-12","timestamp":"2021-02-11T09:47:45.273997074Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9ZVFThqlMn2Lww6c1ZLx1-O","ver":1},"status":{"status":"write"}}},{"id":"1_1613036865.278468_73.1","section":"bucket","name":"tkscyf-12","timestamp":"2021-02-11T09:47:45.278468126Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9ZVFThqlMn2Lww6c1ZLx1-O","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036885.857139_178.1","section":"bucket.instance","name":"tkscyf-13:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.13","timestamp":"2021-02-11T09:48:05.857139297Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LjOnUQz6uMNf3yxGrmeonuy","ver":1},"status":{"status":"write"}}},{"id":"1_1613036885.861989_179.1","section":"bucket.instance","name":"tkscyf-13:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.13","timestamp":"2021-02-11T09:48:05.861989993Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LjOnUQz6uMNf3yxGrmeonuy","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036885.863884_180.1","section":"bucket","name":"tkscyf-13","timestamp":"2021-02-11T09:48:05.863884773Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f0vlqtQxcyVJgs3vWIVlMY5","ver":1},"status":{"status":"write"}}},{"id":"1_1613036885.868524_181.1","section":"bucket","name":"tkscyf-13","timestamp":"2021-02-11T09:48:05.868524702Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f0vlqtQxcyVJgs3vWIVlMY5","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036885.887239_182.1","section":"bucket.instance","name":"tkscyf-14:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.14","timestamp":"2021-02-11T09:48:05.887239288Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Cb1zieCDGGuK7mRxM9X7cOY","ver":1},"status":{"status":"write"}}},{"id":"1_1613036885.892744_183.1","section":"bucket.instance","name":"tkscyf-14:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.14","timestamp":"2021-02-11T09:48:05.892744502Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Cb1zieCDGGuK7mRxM9X7cOY","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036885.894666_184.1","section":"bucket","name":"tkscyf-14","timestamp":"2021-02-11T09:48:05.894666449Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0d8lwgEigMOkq-FSumkqW71","ver":1},"status":{"status":"write"}}},{"id":"1_1613036885.900682_185.1","section":"bucket","name":"tkscyf-14","timestamp":"2021-02-11T09:48:05.900682555Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0d8lwgEigMOkq-FSumkqW71","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036885.942706_186.1","section":"bucket.instance","name":"tkscyf-15:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.15","timestamp":"2021-02-11T09:48:05.942706611Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-hfQZ5lWiy4o6NP7OEZNQ0h","ver":1},"status":{"status":"write"}}},{"id":"1_1613036885.947464_187.1","section":"bucket.instance","name":"tkscyf-15:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.15","timestamp":"2021-02-11T09:48:05.947464351Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-hfQZ5lWiy4o6NP7OEZNQ0h","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036885.949248_188.1","section":"bucket","name":"tkscyf-15","timestamp":"2021-02-11T09:48:05.949248580Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4gdcmHzvr3iqrlpLrZQpyg1","ver":1},"status":{"status":"write"}}},{"id":"1_1613036885.953989_189.1","section":"bucket","name":"tkscyf-15","timestamp":"2021-02-11T09:48:05.953989244Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4gdcmHzvr3iqrlpLrZQpyg1","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036899.854288_194.1","section":"bucket.instance","name":"tkscyf-16:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.16","timestamp":"2021-02-11T09:48:19.854288478Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BjHGVIhMvUslBcZQfe5MTso","ver":1},"status":{"status":"write"}}},{"id":"1_1613036899.859377_195.1","section":"bucket.instance","name":"tkscyf-16:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.16","timestamp":"2021-02-11T09:48:19.859377386Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BjHGVIhMvUslBcZQfe5MTso","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036899.861235_196.1","section":"bucket","name":"tkscyf-16","timestamp":"2021-02-11T09:48:19.861235489Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_mBUX065ZmzTTf0GD1alWPkY","ver":1},"status":{"status":"write"}}},{"id":"1_1613036899.866335_197.1","section":"bucket","name":"tkscyf-16","timestamp":"2021-02-11T09:48:19.866335427Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_mBUX065ZmzTTf0GD1alWPkY","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036899.888709_198.1","section":"bucket.instance","name":"tkscyf-17:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.17","timestamp":"2021-02-11T09:48:19.888709670Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_upQr8yx99-au6_AWkKxYuHu","ver":1},"status":{"status":"write"}}},{"id":"1_1613036899.894272_199.1","section":"bucket.instance","name":"tkscyf-17:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.17","timestamp":"2021-02-11T09:48:19.894272943Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_upQr8yx99-au6_AWkKxYuHu","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036899.896430_200.1","section":"bucket","name":"tkscyf-17","timestamp":"2021-02-11T09:48:19.896430934Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_EVefVFc5TujdFr9n6vuddup","ver":1},"status":{"status":"write"}}},{"id":"1_1613036899.901399_201.1","section":"bucket","name":"tkscyf-17","timestamp":"2021-02-11T09:48:19.901399970Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_EVefVFc5TujdFr9n6vuddup","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036899.942394_202.1","section":"bucket.instance","name":"tkscyf-18:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.18","timestamp":"2021-02-11T09:48:19.942394398Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eLelEHOSgz_5RWVJgejlcwE","ver":1},"status":{"status":"write"}}},{"id":"1_1613036899.947586_203.1","section":"bucket.instance","name":"tkscyf-18:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.18","timestamp":"2021-02-11T09:48:19.947586263Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eLelEHOSgz_5RWVJgejlcwE","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036899.949797_204.1","section":"bucket","name":"tkscyf-18","timestamp":"2021-02-11T09:48:19.949797527Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nnXNYQhXpslLZMkcMsbaLYW","ver":1},"status":{"status":"write"}}},{"id":"1_1613036899.954577_205.1","section":"bucket","name":"tkscyf-18","timestamp":"2021-02-11T09:48:19.954577937Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nnXNYQhXpslLZMkcMsbaLYW","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036899.976313_206.1","section":"bucket.instance","name":"tkscyf-16:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.16","timestamp":"2021-02-11T09:48:19.976313350Z","data":{"read_version":{"tag":"_BjHGVIhMvUslBcZQfe5MTso","ver":1},"write_version":{"tag":"_BjHGVIhMvUslBcZQfe5MTso","ver":2},"status":{"status":"write"}}},{"id":"1_1613036899.981307_207.1","section":"bucket.instance","name":"tkscyf-16:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.16","timestamp":"2021-02-11T09:48:19.981307840Z","data":{"read_version":{"tag":"_BjHGVIhMvUslBcZQfe5MTso","ver":1},"write_version":{"tag":"_BjHGVIhMvUslBcZQfe5MTso","ver":2},"status":{"status":"complete"}}},{"id":"1_1613036899.986228_208.1","section":"bucket.instance","name":"tkscyf-17:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.17","timestamp":"2021-02-11T09:48:19.986228789Z","data":{"read_version":{"tag":"_upQr8yx99-au6_AWkKxYuHu","ver":1},"write_version":{"tag":"_upQr8yx99-au6_AWkKxYuHu","ver":2},"status":{"status":"write"}}},{"id":"1_1613036899.991766_209.1","section":"bucket.instance","name":"tkscyf-17:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.17","timestamp":"2021-02-11T09:48:19.991766410Z","data":{"read_version":{"tag":"_upQr8yx99-au6_AWkKxYuHu","ver":1},"write_version":{"tag":"_upQr8yx99-au6_AWkKxYuHu","ver":2},"status":{"status":"complete"}}},{"id":"1_1613036900.005531_210.1","section":"bucket.instance","name":"tkscyf-18:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.18","timestamp":"2021-02-11T09:48:20.005531886Z","data":{"read_version":{"tag":"_eLelEHOSgz_5RWVJgejlcwE","ver":1},"write_version":{"tag":"_eLelEHOSgz_5RWVJgejlcwE","ver":2},"status":{"status":"write"}}},{"id":"1_1613036900.015316_211.1","section":"bucket.instance","name":"tkscyf-18:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.18","timestamp":"2021-02-11T09:48:20.015316565Z","data":{"read_version":{"tag":"_eLelEHOSgz_5RWVJgejlcwE","ver":1},"write_version":{"tag":"_eLelEHOSgz_5RWVJgejlcwE","ver":2},"status":{"status":"complete"}}},{"id":"1_1613036950.507710_227.1","section":"bucket.instance","name":"tkscyf-19:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.19","timestamp":"2021-02-11T09:49:10.507710607Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_G8JIgZ6gi-tWTxNuGHdxLfV","ver":1},"status":{"status":"write"}}},{"id":"1_1613036950.512728_228.1","section":"bucket.instance","name":"tkscyf-19:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.19","timestamp":"2021-02-11T09:49:10.512728298Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_G8JIgZ6gi-tWTxNuGHdxLfV","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036950.514564_229.1","section":"bucket","name":"tkscyf-19","timestamp":"2021-02-11T09:49:10.514564712Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rUH9m8tDhvujVPRfyQMubOa","ver":1},"status":{"status":"write"}}},{"id":"1_1613036950.519470_230.1","section":"bucket","name":"tkscyf-19","timestamp":"2021-02-11T09:49:10.519470166Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rUH9m8tDhvujVPRfyQMubOa","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036950.525237_231.1","section":"bucket.instance","name":"tkscyf-19:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.19","timestamp":"2021-02-11T09:49:10.525237169Z","data":{"read_version":{"tag":"_G8JIgZ6gi-tWTxNuGHdxLfV","ver":1},"write_version":{"tag":"_G8JIgZ6gi-tWTxNuGHdxLfV","ver":2},"status":{"status":"write"}}},{"id":"1_1613036950.530761_232.1","section":"bucket.instance","name":"tkscyf-19:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.19","timestamp":"2021-02-11T09:49:10.530761661Z","data":{"read_version":{"tag":"_G8JIgZ6gi-tWTxNuGHdxLfV","ver":1},"write_version":{"tag":"_G8JIgZ6gi-tWTxNuGHdxLfV","ver":2},"status":{"status":"complete"}}},{"id":"1_1613036998.267812_238.1","section":"bucket.instance","name":"tkscyf-20:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.20","timestamp":"2021-02-11T09:49:58.267812499Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3X25hzqCxgrngeWBjXAq3MW","ver":1},"status":{"status":"write"}}},{"id":"1_1613036998.273260_239.1","section":"bucket.instance","name":"tkscyf-20:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.20","timestamp":"2021-02-11T09:49:58.273260665Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3X25hzqCxgrngeWBjXAq3MW","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036998.275321_240.1","section":"bucket","name":"tkscyf-20","timestamp":"2021-02-11T09:49:58.275321795Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GSd-2J0lHNWhUfK9gtbzsaz","ver":1},"status":{"status":"write"}}},{"id":"1_1613036998.280111_241.1","section":"bucket","name":"tkscyf-20","timestamp":"2021-02-11T09:49:58.280111713Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GSd-2J0lHNWhUfK9gtbzsaz","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037007.140764_243.1","section":"bucket.instance","name":"tkscyf-20:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.20","timestamp":"2021-02-11T09:50:07.140764026Z","data":{"read_version":{"tag":"_3X25hzqCxgrngeWBjXAq3MW","ver":1},"write_version":{"tag":"_3X25hzqCxgrngeWBjXAq3MW","ver":2},"status":{"status":"write"}}},{"id":"1_1613037007.150589_244.1","section":"bucket.instance","name":"tkscyf-20:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.20","timestamp":"2021-02-11T09:50:07.150589879Z","data":{"read_version":{"tag":"_3X25hzqCxgrngeWBjXAq3MW","ver":1},"write_version":{"tag":"_3X25hzqCxgrngeWBjXAq3MW","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037012.304895_245.1","section":"bucket.instance","name":"tkscyf-20:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.20","timestamp":"2021-02-11T09:50:12.304895189Z","data":{"read_version":{"tag":"_3X25hzqCxgrngeWBjXAq3MW","ver":2},"write_version":{"tag":"_3X25hzqCxgrngeWBjXAq3MW","ver":3},"status":{"status":"write"}}},{"id":"1_1613037012.316265_246.1","section":"bucket.instance","name":"tkscyf-20:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.20","timestamp":"2021-02-11T09:50:12.316265099Z","data":{"read_version":{"tag":"_3X25hzqCxgrngeWBjXAq3MW","ver":2},"write_version":{"tag":"_3X25hzqCxgrngeWBjXAq3MW","ver":3},"status":{"status":"complete"}}},{"id":"1_1613037016.767451_247.1","section":"bucket.instance","name":"tkscyf-21:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.21","timestamp":"2021-02-11T09:50:16.767451636Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_usNhex5LVJs2K5fhCTS-E9B","ver":1},"status":{"status":"write"}}},{"id":"1_1613037016.772791_248.1","section":"bucket.instance","name":"tkscyf-21:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.21","timestamp":"2021-02-11T09:50:16.772791796Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_usNhex5LVJs2K5fhCTS-E9B","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037016.774734_249.1","section":"bucket","name":"tkscyf-21","timestamp":"2021-02-11T09:50:16.774734911Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_oS8fsXNPnQ3YSGsw6yEPI6c","ver":1},"status":{"status":"write"}}},{"id":"1_1613037016.779555_250.1","section":"bucket","name":"tkscyf-21","timestamp":"2021-02-11T09:50:16.779555063Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_oS8fsXNPnQ3YSGsw6yEPI6c","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037016.800629_251.1","section":"bucket.instance","name":"tkscyf-22:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.22","timestamp":"2021-02-11T09:50:16.800629460Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Q5XnU57eMsSQj5AxurjrM1s","ver":1},"status":{"status":"write"}}},{"id":"1_1613037016.807098_252.1","section":"bucket.instance","name":"tkscyf-22:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.22","timestamp":"2021-02-11T09:50:16.807098120Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Q5XnU57eMsSQj5AxurjrM1s","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037016.809071_253.1","section":"bucket","name":"tkscyf-22","timestamp":"2021-02-11T09:50:16.809071480Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Jcw-M7gbUwOe2t495PGK9vo","ver":1},"status":{"status":"write"}}},{"id":"1_1613037016.813672_254.1","section":"bucket","name":"tkscyf-22","timestamp":"2021-02-11T09:50:16.813672741Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Jcw-M7gbUwOe2t495PGK9vo","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037016.860072_255.1","section":"bucket.instance","name":"tkscyf-23:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.23","timestamp":"2021-02-11T09:50:16.860072689Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GCLr8VFsBOKFg5ZuZUx_GvZ","ver":1},"status":{"status":"write"}}},{"id":"1_1613037016.866566_256.1","section":"bucket.instance","name":"tkscyf-23:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.23","timestamp":"2021-02-11T09:50:16.866566184Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GCLr8VFsBOKFg5ZuZUx_GvZ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037016.869661_257.1","section":"bucket","name":"tkscyf-23","timestamp":"2021-02-11T09:50:16.869661986Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8HFWSnbWVi7UHPWj_xF4vXf","ver":1},"status":{"status":"write"}}},{"id":"1_1613037016.879001_258.1","section":"bucket","name":"tkscyf-23","timestamp":"2021-02-11T09:50:16.879001731Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8HFWSnbWVi7UHPWj_xF4vXf","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037016.897902_259.1","section":"bucket.instance","name":"tkscyf-21:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.21","timestamp":"2021-02-11T09:50:16.897902097Z","data":{"read_version":{"tag":"_usNhex5LVJs2K5fhCTS-E9B","ver":1},"write_version":{"tag":"_usNhex5LVJs2K5fhCTS-E9B","ver":2},"status":{"status":"write"}}},{"id":"1_1613037016.902295_260.1","section":"bucket.instance","name":"tkscyf-21:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.21","timestamp":"2021-02-11T09:50:16.902295553Z","data":{"read_version":{"tag":"_usNhex5LVJs2K5fhCTS-E9B","ver":1},"write_version":{"tag":"_usNhex5LVJs2K5fhCTS-E9B","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037016.906251_261.1","section":"bucket.instance","name":"tkscyf-22:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.22","timestamp":"2021-02-11T09:50:16.906251476Z","data":{"read_version":{"tag":"_Q5XnU57eMsSQj5AxurjrM1s","ver":1},"write_version":{"tag":"_Q5XnU57eMsSQj5AxurjrM1s","ver":2},"status":{"status":"write"}}},{"id":"1_1613037016.911598_262.1","section":"bucket.instance","name":"tkscyf-22:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.22","timestamp":"2021-02-11T09:50:16.911598777Z","data":{"read_version":{"tag":"_Q5XnU57eMsSQj5AxurjrM1s","ver":1},"write_version":{"tag":"_Q5XnU57eMsSQj5AxurjrM1s","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037016.926597_263.1","section":"bucket.instance","name":"tkscyf-23:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.23","timestamp":"2021-02-11T09:50:16.926597890Z","data":{"read_version":{"tag":"_GCLr8VFsBOKFg5ZuZUx_GvZ","ver":1},"write_version":{"tag":"_GCLr8VFsBOKFg5ZuZUx_GvZ","ver":2},"status":{"status":"write"}}},{"id":"1_1613037016.931312_264.1","section":"bucket.instance","name":"tkscyf-23:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.23","timestamp":"2021-02-11T09:50:16.931312694Z","data":{"read_version":{"tag":"_GCLr8VFsBOKFg5ZuZUx_GvZ","ver":1},"write_version":{"tag":"_GCLr8VFsBOKFg5ZuZUx_GvZ","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037035.238424_265.1","section":"bucket.instance","name":"tkscyf-24:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.24","timestamp":"2021-02-11T09:50:35.238424865Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bb0kUiSwNRO-U3TZ4E1z9m5","ver":1},"status":{"status":"write"}}},{"id":"1_1613037035.243640_266.1","section":"bucket.instance","name":"tkscyf-24:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.24","timestamp":"2021-02-11T09:50:35.243640241Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bb0kUiSwNRO-U3TZ4E1z9m5","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037035.245475_267.1","section":"bucket","name":"tkscyf-24","timestamp":"2021-02-11T09:50:35.245475848Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7Yg_eLnx93F7TXgKcKQAxT5","ver":1},"status":{"status":"write"}}},{"id":"1_1613037035.250290_268.1","section":"bucket","name":"tkscyf-24","timestamp":"2021-02-11T09:50:35.250290672Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7Yg_eLnx93F7TXgKcKQAxT5","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037035.274261_269.1","section":"bucket.instance","name":"tkscyf-25:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.25","timestamp":"2021-02-11T09:50:35.274261140Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1nnotJE0A2l-djRVSBIWs9E","ver":1},"status":{"status":"write"}}},{"id":"1_1613037035.281072_270.1","section":"bucket.instance","name":"tkscyf-25:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.25","timestamp":"2021-02-11T09:50:35.281072011Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1nnotJE0A2l-djRVSBIWs9E","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037035.283085_271.1","section":"bucket","name":"tkscyf-25","timestamp":"2021-02-11T09:50:35.283085981Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PBWPxzfNM3rpuuSRbw9WWJ4","ver":1},"status":{"status":"write"}}},{"id":"1_1613037035.287331_272.1","section":"bucket","name":"tkscyf-25","timestamp":"2021-02-11T09:50:35.287331936Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PBWPxzfNM3rpuuSRbw9WWJ4","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037035.327494_273.1","section":"bucket.instance","name":"tkscyf-26:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.26","timestamp":"2021-02-11T09:50:35.327494695Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yBW2OnCxamt6jwtfytnSIYp","ver":1},"status":{"status":"write"}}},{"id":"1_1613037035.336740_274.1","section":"bucket.instance","name":"tkscyf-26:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.26","timestamp":"2021-02-11T09:50:35.336740442Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yBW2OnCxamt6jwtfytnSIYp","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037035.338820_275.1","section":"bucket","name":"tkscyf-26","timestamp":"2021-02-11T09:50:35.338820669Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GBS2JL6Meg0eeenzd3LFcRc","ver":1},"status":{"status":"write"}}},{"id":"1_1613037035.343318_276.1","section":"bucket","name":"tkscyf-26","timestamp":"2021-02-11T09:50:35.343318410Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GBS2JL6Meg0eeenzd3LFcRc","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037035.361989_277.1","section":"bucket.instance","name":"tkscyf-24:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.24","timestamp":"2021-02-11T09:50:35.361989484Z","data":{"read_version":{"tag":"_bb0kUiSwNRO-U3TZ4E1z9m5","ver":1},"write_version":{"tag":"_bb0kUiSwNRO-U3TZ4E1z9m5","ver":2},"status":{"status":"write"}}},{"id":"1_1613037035.367759_278.1","section":"bucket.instance","name":"tkscyf-24:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.24","timestamp":"2021-02-11T09:50:35.367759053Z","data":{"read_version":{"tag":"_bb0kUiSwNRO-U3TZ4E1z9m5","ver":1},"write_version":{"tag":"_bb0kUiSwNRO-U3TZ4E1z9m5","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037035.371457_279.1","section":"bucket.instance","name":"tkscyf-24:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.24","timestamp":"2021-02-11T09:50:35.371457040Z","data":{"read_version":{"tag":"_bb0kUiSwNRO-U3TZ4E1z9m5","ver":2},"write_version":{"tag":"_bb0kUiSwNRO-U3TZ4E1z9m5","ver":3},"status":{"status":"write"}}},{"id":"1_1613037035.376968_280.1","section":"bucket.instance","name":"tkscyf-24:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.24","timestamp":"2021-02-11T09:50:35.376968483Z","data":{"read_version":{"tag":"_bb0kUiSwNRO-U3TZ4E1z9m5","ver":2},"write_version":{"tag":"_bb0kUiSwNRO-U3TZ4E1z9m5","ver":3},"status":{"status":"complete"}}},{"id":"1_1613037035.382067_281.1","section":"bucket.instance","name":"tkscyf-25:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.25","timestamp":"2021-02-11T09:50:35.382067509Z","data":{"read_version":{"tag":"_1nnotJE0A2l-djRVSBIWs9E","ver":1},"write_version":{"tag":"_1nnotJE0A2l-djRVSBIWs9E","ver":2},"status":{"status":"write"}}},{"id":"1_1613037035.388230_282.1","section":"bucket.instance","name":"tkscyf-25:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.25","timestamp":"2021-02-11T09:50:35.388230116Z","data":{"read_version":{"tag":"_1nnotJE0A2l-djRVSBIWs9E","ver":1},"write_version":{"tag":"_1nnotJE0A2l-djRVSBIWs9E","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037035.399024_283.1","section":"bucket.instance","name":"tkscyf-25:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.25","timestamp":"2021-02-11T09:50:35.399024419Z","data":{"read_version":{"tag":"_1nnotJE0A2l-djRVSBIWs9E","ver":2},"write_version":{"tag":"_1nnotJE0A2l-djRVSBIWs9E","ver":3},"status":{"status":"write"}}},{"id":"1_1613037035.406913_284.1","section":"bucket.instance","name":"tkscyf-25:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.25","timestamp":"2021-02-11T09:50:35.406913878Z","data":{"read_version":{"tag":"_1nnotJE0A2l-djRVSBIWs9E","ver":2},"write_version":{"tag":"_1nnotJE0A2l-djRVSBIWs9E","ver":3},"status":{"status":"complete"}}},{"id":"1_1613037035.418911_285.1","section":"bucket.instance","name":"tkscyf-26:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.26","timestamp":"2021-02-11T09:50:35.418911493Z","data":{"read_version":{"tag":"_yBW2OnCxamt6jwtfytnSIYp","ver":1},"write_version":{"tag":"_yBW2OnCxamt6jwtfytnSIYp","ver":2},"status":{"status":"write"}}},{"id":"1_1613037035.427825_286.1","section":"bucket.instance","name":"tkscyf-26:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.26","timestamp":"2021-02-11T09:50:35.427825538Z","data":{"read_version":{"tag":"_yBW2OnCxamt6jwtfytnSIYp","ver":1},"write_version":{"tag":"_yBW2OnCxamt6jwtfytnSIYp","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037035.439773_287.1","section":"bucket.instance","name":"tkscyf-26:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.26","timestamp":"2021-02-11T09:50:35.439773122Z","data":{"read_version":{"tag":"_yBW2OnCxamt6jwtfytnSIYp","ver":2},"write_version":{"tag":"_yBW2OnCxamt6jwtfytnSIYp","ver":3},"status":{"status":"write"}}},{"id":"1_1613037035.445782_288.1","section":"bucket.instance","name":"tkscyf-26:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.26","timestamp":"2021-02-11T09:50:35.445782025Z","data":{"read_version":{"tag":"_yBW2OnCxamt6jwtfytnSIYp","ver":2},"write_version":{"tag":"_yBW2OnCxamt6jwtfytnSIYp","ver":3},"status":{"status":"complete"}}},{"id":"1_1613037049.014616_289.1","section":"bucket.instance","name":"tkscyf-27:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.27","timestamp":"2021-02-11T09:50:49.014616484Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HPEAummOtXzD3iTYGsjL78L","ver":1},"status":{"status":"write"}}},{"id":"1_1613037049.020486_290.1","section":"bucket.instance","name":"tkscyf-27:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.27","timestamp":"2021-02-11T09:50:49.020486409Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HPEAummOtXzD3iTYGsjL78L","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037049.022446_291.1","section":"bucket","name":"tkscyf-27","timestamp":"2021-02-11T09:50:49.022446056Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HzrXfJNvB4DsZqVjRjsmkDL","ver":1},"status":{"status":"write"}}},{"id":"1_1613037049.026934_292.1","section":"bucket","name":"tkscyf-27","timestamp":"2021-02-11T09:50:49.026934689Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HzrXfJNvB4DsZqVjRjsmkDL","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037049.052544_293.1","section":"bucket.instance","name":"tkscyf-28:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.28","timestamp":"2021-02-11T09:50:49.052544751Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-VSdKrlRKUfuwloTfMI6Xuv","ver":1},"status":{"status":"write"}}},{"id":"1_1613037049.057739_294.1","section":"bucket.instance","name":"tkscyf-28:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.28","timestamp":"2021-02-11T09:50:49.057739941Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-VSdKrlRKUfuwloTfMI6Xuv","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037049.059973_295.1","section":"bucket","name":"tkscyf-28","timestamp":"2021-02-11T09:50:49.059973620Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-BX9bB1FIeOQc9AK3EsxPMy","ver":1},"status":{"status":"write"}}},{"id":"1_1613037049.064978_296.1","section":"bucket","name":"tkscyf-28","timestamp":"2021-02-11T09:50:49.064978209Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-BX9bB1FIeOQc9AK3EsxPMy","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037049.129163_297.1","section":"bucket.instance","name":"tkscyf-29:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.29","timestamp":"2021-02-11T09:50:49.129163449Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hyJCgZ3ZqhjZ2B82NLQtbFj","ver":1},"status":{"status":"write"}}},{"id":"1_1613037049.139199_298.1","section":"bucket.instance","name":"tkscyf-29:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.29","timestamp":"2021-02-11T09:50:49.139199855Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hyJCgZ3ZqhjZ2B82NLQtbFj","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037049.141005_299.1","section":"bucket","name":"tkscyf-29","timestamp":"2021-02-11T09:50:49.141005489Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1eRc33iNyGdkXUg6pUKi_qn","ver":1},"status":{"status":"write"}}},{"id":"1_1613037049.145434_300.1","section":"bucket","name":"tkscyf-29","timestamp":"2021-02-11T09:50:49.145434314Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1eRc33iNyGdkXUg6pUKi_qn","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037049.166096_301.1","section":"bucket.instance","name":"tkscyf-27:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.27","timestamp":"2021-02-11T09:50:49.166096762Z","data":{"read_version":{"tag":"_HPEAummOtXzD3iTYGsjL78L","ver":1},"write_version":{"tag":"_HPEAummOtXzD3iTYGsjL78L","ver":2},"status":{"status":"write"}}},{"id":"1_1613037049.171244_302.1","section":"bucket.instance","name":"tkscyf-27:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.27","timestamp":"2021-02-11T09:50:49.171244329Z","data":{"read_version":{"tag":"_HPEAummOtXzD3iTYGsjL78L","ver":1},"write_version":{"tag":"_HPEAummOtXzD3iTYGsjL78L","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037049.219869_303.1","section":"bucket.instance","name":"tkscyf-28:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.28","timestamp":"2021-02-11T09:50:49.219869180Z","data":{"read_version":{"tag":"_-VSdKrlRKUfuwloTfMI6Xuv","ver":1},"write_version":{"tag":"_-VSdKrlRKUfuwloTfMI6Xuv","ver":2},"status":{"status":"write"}}},{"id":"1_1613037049.230173_304.1","section":"bucket.instance","name":"tkscyf-28:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.28","timestamp":"2021-02-11T09:50:49.230173871Z","data":{"read_version":{"tag":"_-VSdKrlRKUfuwloTfMI6Xuv","ver":1},"write_version":{"tag":"_-VSdKrlRKUfuwloTfMI6Xuv","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037049.287108_305.1","section":"bucket.instance","name":"tkscyf-29:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.29","timestamp":"2021-02-11T09:50:49.287108951Z","data":{"read_version":{"tag":"_hyJCgZ3ZqhjZ2B82NLQtbFj","ver":1},"write_version":{"tag":"_hyJCgZ3ZqhjZ2B82NLQtbFj","ver":2},"status":{"status":"write"}}},{"id":"1_1613037049.300847_306.1","section":"bucket.instance","name":"tkscyf-29:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.29","timestamp":"2021-02-11T09:50:49.300847185Z","data":{"read_version":{"tag":"_hyJCgZ3ZqhjZ2B82NLQtbFj","ver":1},"write_version":{"tag":"_hyJCgZ3ZqhjZ2B82NLQtbFj","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037049.364299_307.1","section":"bucket.instance","name":"tkscyf-30:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.30","timestamp":"2021-02-11T09:50:49.364299749Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Es0JugEkT-f3o-riZAdqnrS","ver":1},"status":{"status":"write"}}},{"id":"1_1613037049.370623_308.1","section":"bucket.instance","name":"tkscyf-30:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.30","timestamp":"2021-02-11T09:50:49.370623916Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Es0JugEkT-f3o-riZAdqnrS","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037049.372900_309.1","section":"bucket","name":"tkscyf-30","timestamp":"2021-02-11T09:50:49.372900977Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0ipPDFF-oXK1PAXV9ImeRz8","ver":1},"status":{"status":"write"}}},{"id":"1_1613037049.377959_310.1","section":"bucket","name":"tkscyf-30","timestamp":"2021-02-11T09:50:49.377959725Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0ipPDFF-oXK1PAXV9ImeRz8","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037049.400272_311.1","section":"bucket.instance","name":"tkscyf-31:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.31","timestamp":"2021-02-11T09:50:49.400272843Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_J_JeLJIs-yKAxn3I-jH6tun","ver":1},"status":{"status":"write"}}},{"id":"1_1613037049.406204_312.1","section":"bucket.instance","name":"tkscyf-31:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.31","timestamp":"2021-02-11T09:50:49.406204831Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_J_JeLJIs-yKAxn3I-jH6tun","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037049.407984_313.1","section":"bucket","name":"tkscyf-31","timestamp":"2021-02-11T09:50:49.407984579Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AtSnm2w_MDdPl4ZTuW8BBX1","ver":1},"status":{"status":"write"}}},{"id":"1_1613037049.412626_314.1","section":"bucket","name":"tkscyf-31","timestamp":"2021-02-11T09:50:49.412626629Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AtSnm2w_MDdPl4ZTuW8BBX1","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037049.456098_315.1","section":"bucket.instance","name":"tkscyf-32:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.32","timestamp":"2021-02-11T09:50:49.456098996Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jg7YdCRiL4_wX7QgQ9ZEBCK","ver":1},"status":{"status":"write"}}},{"id":"1_1613037049.461384_316.1","section":"bucket.instance","name":"tkscyf-32:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.32","timestamp":"2021-02-11T09:50:49.461384431Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jg7YdCRiL4_wX7QgQ9ZEBCK","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037049.463339_317.1","section":"bucket","name":"tkscyf-32","timestamp":"2021-02-11T09:50:49.463339008Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eg1axhWTHrzA2NWrcdnReCD","ver":1},"status":{"status":"write"}}},{"id":"1_1613037049.468298_318.1","section":"bucket","name":"tkscyf-32","timestamp":"2021-02-11T09:50:49.468298585Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eg1axhWTHrzA2NWrcdnReCD","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037049.493719_319.1","section":"bucket.instance","name":"tkscyf-30:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.30","timestamp":"2021-02-11T09:50:49.493719070Z","data":{"read_version":{"tag":"_Es0JugEkT-f3o-riZAdqnrS","ver":1},"write_version":{"tag":"_Es0JugEkT-f3o-riZAdqnrS","ver":2},"status":{"status":"write"}}},{"id":"1_1613037049.498976_320.1","section":"bucket.instance","name":"tkscyf-30:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.30","timestamp":"2021-02-11T09:50:49.498976233Z","data":{"read_version":{"tag":"_Es0JugEkT-f3o-riZAdqnrS","ver":1},"write_version":{"tag":"_Es0JugEkT-f3o-riZAdqnrS","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037049.547179_321.1","section":"bucket.instance","name":"tkscyf-31:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.31","timestamp":"2021-02-11T09:50:49.547179938Z","data":{"read_version":{"tag":"_J_JeLJIs-yKAxn3I-jH6tun","ver":1},"write_version":{"tag":"_J_JeLJIs-yKAxn3I-jH6tun","ver":2},"status":{"status":"write"}}},{"id":"1_1613037049.552155_322.1","section":"bucket.instance","name":"tkscyf-31:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.31","timestamp":"2021-02-11T09:50:49.552155032Z","data":{"read_version":{"tag":"_J_JeLJIs-yKAxn3I-jH6tun","ver":1},"write_version":{"tag":"_J_JeLJIs-yKAxn3I-jH6tun","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037049.603090_323.1","section":"bucket.instance","name":"tkscyf-32:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.32","timestamp":"2021-02-11T09:50:49.603090607Z","data":{"read_version":{"tag":"_jg7YdCRiL4_wX7QgQ9ZEBCK","ver":1},"write_version":{"tag":"_jg7YdCRiL4_wX7QgQ9ZEBCK","ver":2},"status":{"status":"write"}}},{"id":"1_1613037049.608976_324.1","section":"bucket.instance","name":"tkscyf-32:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.32","timestamp":"2021-02-11T09:50:49.608976182Z","data":{"read_version":{"tag":"_jg7YdCRiL4_wX7QgQ9ZEBCK","ver":1},"write_version":{"tag":"_jg7YdCRiL4_wX7QgQ9ZEBCK","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037049.680125_325.1","section":"bucket.instance","name":"tkscyf-33:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.33","timestamp":"2021-02-11T09:50:49.680125618Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eZRAP_8ajxLAWs4hu7nqOwz","ver":1},"status":{"status":"write"}}},{"id":"1_1613037049.687075_326.1","section":"bucket.instance","name":"tkscyf-33:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.33","timestamp":"2021-02-11T09:50:49.687075481Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eZRAP_8ajxLAWs4hu7nqOwz","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037049.692386_327.1","section":"bucket","name":"tkscyf-33","timestamp":"2021-02-11T09:50:49.692386405Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-6dR6mEXl2aMVHaPbOLG9sm","ver":1},"status":{"status":"write"}}},{"id":"1_1613037049.699844_328.1","section":"bucket","name":"tkscyf-33","timestamp":"2021-02-11T09:50:49.699844848Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-6dR6mEXl2aMVHaPbOLG9sm","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037049.725396_329.1","section":"bucket.instance","name":"tkscyf-34:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.34","timestamp":"2021-02-11T09:50:49.725396604Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_k7kM2SM39DqWgu3NfoI4bqF","ver":1},"status":{"status":"write"}}},{"id":"1_1613037049.731707_330.1","section":"bucket.instance","name":"tkscyf-34:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.34","timestamp":"2021-02-11T09:50:49.731707348Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_k7kM2SM39DqWgu3NfoI4bqF","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037049.735696_331.1","section":"bucket","name":"tkscyf-34","timestamp":"2021-02-11T09:50:49.735696647Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_L016eB3Fp2Gc1WBb3ihrD9w","ver":1},"status":{"status":"write"}}},{"id":"1_1613037049.740587_332.1","section":"bucket","name":"tkscyf-34","timestamp":"2021-02-11T09:50:49.740587087Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_L016eB3Fp2Gc1WBb3ihrD9w","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037049.780026_333.1","section":"bucket.instance","name":"tkscyf-35:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.35","timestamp":"2021-02-11T09:50:49.780026803Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KosuwqaTc-yYnhDuNK50-jc","ver":1},"status":{"status":"write"}}},{"id":"1_1613037049.785523_334.1","section":"bucket.instance","name":"tkscyf-35:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.35","timestamp":"2021-02-11T09:50:49.785523728Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KosuwqaTc-yYnhDuNK50-jc","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037049.787540_335.1","section":"bucket","name":"tkscyf-35","timestamp":"2021-02-11T09:50:49.787540619Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wBCuU48Ox_uSnLpuXpSWi95","ver":1},"status":{"status":"write"}}},{"id":"1_1613037049.792438_336.1","section":"bucket","name":"tkscyf-35","timestamp":"2021-02-11T09:50:49.792438114Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wBCuU48Ox_uSnLpuXpSWi95","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037049.814263_337.1","section":"bucket.instance","name":"tkscyf-33:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.33","timestamp":"2021-02-11T09:50:49.814263352Z","data":{"read_version":{"tag":"_eZRAP_8ajxLAWs4hu7nqOwz","ver":1},"write_version":{"tag":"_eZRAP_8ajxLAWs4hu7nqOwz","ver":2},"status":{"status":"write"}}},{"id":"1_1613037049.819262_338.1","section":"bucket.instance","name":"tkscyf-33:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.33","timestamp":"2021-02-11T09:50:49.819262498Z","data":{"read_version":{"tag":"_eZRAP_8ajxLAWs4hu7nqOwz","ver":1},"write_version":{"tag":"_eZRAP_8ajxLAWs4hu7nqOwz","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037050.867169_339.1","section":"bucket.instance","name":"tkscyf-34:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.34","timestamp":"2021-02-11T09:50:50.867169930Z","data":{"read_version":{"tag":"_k7kM2SM39DqWgu3NfoI4bqF","ver":1},"write_version":{"tag":"_k7kM2SM39DqWgu3NfoI4bqF","ver":2},"status":{"status":"write"}}},{"id":"1_1613037050.879211_340.1","section":"bucket.instance","name":"tkscyf-34:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.34","timestamp":"2021-02-11T09:50:50.879211406Z","data":{"read_version":{"tag":"_k7kM2SM39DqWgu3NfoI4bqF","ver":1},"write_version":{"tag":"_k7kM2SM39DqWgu3NfoI4bqF","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037051.935225_341.1","section":"bucket.instance","name":"tkscyf-35:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.35","timestamp":"2021-02-11T09:50:51.935225906Z","data":{"read_version":{"tag":"_KosuwqaTc-yYnhDuNK50-jc","ver":1},"write_version":{"tag":"_KosuwqaTc-yYnhDuNK50-jc","ver":2},"status":{"status":"write"}}},{"id":"1_1613037051.946001_342.1","section":"bucket.instance","name":"tkscyf-35:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.35","timestamp":"2021-02-11T09:50:51.946001361Z","data":{"read_version":{"tag":"_KosuwqaTc-yYnhDuNK50-jc","ver":1},"write_version":{"tag":"_KosuwqaTc-yYnhDuNK50-jc","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037056.396292_343.1","section":"bucket.instance","name":"tkscyf-36:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.36","timestamp":"2021-02-11T09:50:56.396292831Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ssZsgWPzFUlbtHH72rboNt2","ver":1},"status":{"status":"write"}}},{"id":"1_1613037056.402315_344.1","section":"bucket.instance","name":"tkscyf-36:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.36","timestamp":"2021-02-11T09:50:56.402315663Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ssZsgWPzFUlbtHH72rboNt2","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037056.404168_345.1","section":"bucket","name":"tkscyf-36","timestamp":"2021-02-11T09:50:56.404168283Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_R9PAwh1OvxD1UjEPuuj8qio","ver":1},"status":{"status":"write"}}},{"id":"1_1613037056.408811_346.1","section":"bucket","name":"tkscyf-36","timestamp":"2021-02-11T09:50:56.408811301Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_R9PAwh1OvxD1UjEPuuj8qio","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037056.437739_347.1","section":"bucket.instance","name":"tkscyf-37:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.37","timestamp":"2021-02-11T09:50:56.437739913Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HWaNKKBo9OCATJepv1PpQ9b","ver":1},"status":{"status":"write"}}},{"id":"1_1613037056.442573_348.1","section":"bucket.instance","name":"tkscyf-37:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.37","timestamp":"2021-02-11T09:50:56.442573229Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HWaNKKBo9OCATJepv1PpQ9b","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037056.444396_349.1","section":"bucket","name":"tkscyf-37","timestamp":"2021-02-11T09:50:56.444396764Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RbeSoBdgaTvi80yVezMNU9a","ver":1},"status":{"status":"write"}}},{"id":"1_1613037056.449127_350.1","section":"bucket","name":"tkscyf-37","timestamp":"2021-02-11T09:50:56.449127942Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RbeSoBdgaTvi80yVezMNU9a","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037056.503850_351.1","section":"bucket.instance","name":"tkscyf-38:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.38","timestamp":"2021-02-11T09:50:56.503850524Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_M7gYPbaihLb1WY3hhZTdAtg","ver":1},"status":{"status":"write"}}},{"id":"1_1613037056.508532_352.1","section":"bucket.instance","name":"tkscyf-38:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.38","timestamp":"2021-02-11T09:50:56.508532698Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_M7gYPbaihLb1WY3hhZTdAtg","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037056.510275_353.1","section":"bucket","name":"tkscyf-38","timestamp":"2021-02-11T09:50:56.510275102Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0RzXHcbVo20pBILfHDUWraY","ver":1},"status":{"status":"write"}}},{"id":"1_1613037056.514482_354.1","section":"bucket","name":"tkscyf-38","timestamp":"2021-02-11T09:50:56.514482141Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0RzXHcbVo20pBILfHDUWraY","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037057.908443_355.1","section":"bucket.instance","name":"tkscyf-39:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.39","timestamp":"2021-02-11T09:50:57.908443576Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_94HMkmvhzlZXLQvruAOVtAJ","ver":1},"status":{"status":"write"}}},{"id":"1_1613037057.914397_356.1","section":"bucket.instance","name":"tkscyf-39:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.39","timestamp":"2021-02-11T09:50:57.914397041Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_94HMkmvhzlZXLQvruAOVtAJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037057.916715_357.1","section":"bucket","name":"tkscyf-39","timestamp":"2021-02-11T09:50:57.916715595Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wsKJ1QOckGXWesrqOCyo1WG","ver":1},"status":{"status":"write"}}},{"id":"1_1613037057.923356_358.1","section":"bucket","name":"tkscyf-39","timestamp":"2021-02-11T09:50:57.923356621Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wsKJ1QOckGXWesrqOCyo1WG","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037057.945478_359.1","section":"bucket.instance","name":"tkscyf-40:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.40","timestamp":"2021-02-11T09:50:57.945478017Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pg3ahh-4pO-_3XyceN-gWsQ","ver":1},"status":{"status":"write"}}},{"id":"1_1613037057.949753_360.1","section":"bucket.instance","name":"tkscyf-40:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.40","timestamp":"2021-02-11T09:50:57.949753049Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pg3ahh-4pO-_3XyceN-gWsQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037057.951444_361.1","section":"bucket","name":"tkscyf-40","timestamp":"2021-02-11T09:50:57.951444977Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ej1h_jFuYjzyoRvb-1C7RAf","ver":1},"status":{"status":"write"}}},{"id":"1_1613037057.955584_362.1","section":"bucket","name":"tkscyf-40","timestamp":"2021-02-11T09:50:57.955584939Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ej1h_jFuYjzyoRvb-1C7RAf","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037057.996014_363.1","section":"bucket.instance","name":"tkscyf-41:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.41","timestamp":"2021-02-11T09:50:57.996014692Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_I9DAcVu7OInplRV1Ta3wruY","ver":1},"status":{"status":"write"}}},{"id":"1_1613037058.000113_364.1","section":"bucket.instance","name":"tkscyf-41:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.41","timestamp":"2021-02-11T09:50:58.000113498Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_I9DAcVu7OInplRV1Ta3wruY","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037058.001757_365.1","section":"bucket","name":"tkscyf-41","timestamp":"2021-02-11T09:50:58.001757095Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_R0j0lRoRH3ZszkT7rysin2m","ver":1},"status":{"status":"write"}}},{"id":"1_1613037058.005761_366.1","section":"bucket","name":"tkscyf-41","timestamp":"2021-02-11T09:50:58.005761416Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_R0j0lRoRH3ZszkT7rysin2m","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036843.308060_40.1","section":"bucket.instance","name":"tkscyf-1:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.1","timestamp":"2021-02-11T09:47:23.308060874Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vJcQgK-3XDRzzGYAO1_aHfA","ver":1},"status":{"status":"write"}}},{"id":"1_1613036843.334677_41.1","section":"bucket.instance","name":"tkscyf-1:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.1","timestamp":"2021-02-11T09:47:23.334677253Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vJcQgK-3XDRzzGYAO1_aHfA","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036843.341061_42.1","section":"bucket","name":"tkscyf-1","timestamp":"2021-02-11T09:47:23.341061810Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_so1GMpfrEZgmi_sTYZSAr8H","ver":1},"status":{"status":"write"}}},{"id":"1_1613036843.345437_43.1","section":"bucket","name":"tkscyf-1","timestamp":"2021-02-11T09:47:23.345437347Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_so1GMpfrEZgmi_sTYZSAr8H","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036843.366232_44.1","section":"bucket.instance","name":"tkscyf-2:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.2","timestamp":"2021-02-11T09:47:23.366232598Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MZ24WdPDpXrrqQPnafXRqiN","ver":1},"status":{"status":"write"}}},{"id":"1_1613036843.371311_45.1","section":"bucket.instance","name":"tkscyf-2:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.2","timestamp":"2021-02-11T09:47:23.371311720Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MZ24WdPDpXrrqQPnafXRqiN","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036843.373088_46.1","section":"bucket","name":"tkscyf-2","timestamp":"2021-02-11T09:47:23.373088017Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5VYuSj_-KrHdRO5ccn8Xe_y","ver":1},"status":{"status":"write"}}},{"id":"1_1613036843.378050_47.1","section":"bucket","name":"tkscyf-2","timestamp":"2021-02-11T09:47:23.378050996Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5VYuSj_-KrHdRO5ccn8Xe_y","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036847.401028_48.1","section":"bucket.instance","name":"tkscyf-3:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.3","timestamp":"2021-02-11T09:47:27.401028055Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ymULWMaifA40PYB81mtiqKO","ver":1},"status":{"status":"write"}}},{"id":"1_1613036847.406630_49.1","section":"bucket.instance","name":"tkscyf-3:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.3","timestamp":"2021-02-11T09:47:27.406630738Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ymULWMaifA40PYB81mtiqKO","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036847.408599_50.1","section":"bucket","name":"tkscyf-3","timestamp":"2021-02-11T09:47:27.408599295Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8F2ONXee867moT-prh9FMvz","ver":1},"status":{"status":"write"}}},{"id":"1_1613036847.414041_51.1","section":"bucket","name":"tkscyf-3","timestamp":"2021-02-11T09:47:27.414041891Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8F2ONXee867moT-prh9FMvz","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036848.515900_52.1","section":"bucket.instance","name":"tkscyf-4:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.4","timestamp":"2021-02-11T09:47:28.515900958Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OC0h4nNzRptiVogF5DDSlbl","ver":1},"status":{"status":"write"}}},{"id":"1_1613036848.521086_53.1","section":"bucket.instance","name":"tkscyf-4:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.4","timestamp":"2021-02-11T09:47:28.521086371Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OC0h4nNzRptiVogF5DDSlbl","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036848.523142_54.1","section":"bucket","name":"tkscyf-4","timestamp":"2021-02-11T09:47:28.523142343Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lzBpaQfMhBTasrckHQnxsrH","ver":1},"status":{"status":"write"}}},{"id":"1_1613036848.528085_55.1","section":"bucket","name":"tkscyf-4","timestamp":"2021-02-11T09:47:28.528085857Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lzBpaQfMhBTasrckHQnxsrH","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036848.545712_56.1","section":"bucket.instance","name":"tkscyf-5:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.5","timestamp":"2021-02-11T09:47:28.545712519Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pizM_4Vxb6ga5d7zQm5cup4","ver":1},"status":{"status":"write"}}},{"id":"1_1613036848.551316_57.1","section":"bucket.instance","name":"tkscyf-5:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.5","timestamp":"2021-02-11T09:47:28.551316982Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pizM_4Vxb6ga5d7zQm5cup4","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036848.553108_58.1","section":"bucket","name":"tkscyf-5","timestamp":"2021-02-11T09:47:28.553108370Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__btqQaGnBiVqesmRij4KpZu","ver":1},"status":{"status":"write"}}},{"id":"1_1613036848.557871_59.1","section":"bucket","name":"tkscyf-5","timestamp":"2021-02-11T09:47:28.557871264Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__btqQaGnBiVqesmRij4KpZu","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036848.598426_60.1","section":"bucket.instance","name":"tkscyf-6:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.6","timestamp":"2021-02-11T09:47:28.598426655Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MXFYvcf4zPQLqC1AyVkpK27","ver":1},"status":{"status":"write"}}},{"id":"1_1613036848.607442_61.1","section":"bucket.instance","name":"tkscyf-6:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.6","timestamp":"2021-02-11T09:47:28.607442762Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MXFYvcf4zPQLqC1AyVkpK27","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036848.609097_62.1","section":"bucket","name":"tkscyf-6","timestamp":"2021-02-11T09:47:28.609097002Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_27D4N2MBNgnn-CUkGMJjydq","ver":1},"status":{"status":"write"}}},{"id":"1_1613036848.613340_63.1","section":"bucket","name":"tkscyf-6","timestamp":"2021-02-11T09:47:28.613340684Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_27D4N2MBNgnn-CUkGMJjydq","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036851.051651_64.1","section":"bucket.instance","name":"tkscyf-7:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.7","timestamp":"2021-02-11T09:47:31.051651058Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qdftsvlxuD5g0Vn1ajzNmyo","ver":1},"status":{"status":"write"}}},{"id":"1_1613036851.056767_65.1","section":"bucket.instance","name":"tkscyf-7:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.7","timestamp":"2021-02-11T09:47:31.056767922Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qdftsvlxuD5g0Vn1ajzNmyo","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036851.058594_66.1","section":"bucket","name":"tkscyf-7","timestamp":"2021-02-11T09:47:31.058594967Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FCpy5wrdCKP58fp9iJu68WQ","ver":1},"status":{"status":"write"}}},{"id":"1_1613036851.063396_67.1","section":"bucket","name":"tkscyf-7","timestamp":"2021-02-11T09:47:31.063396372Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FCpy5wrdCKP58fp9iJu68WQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036851.082324_68.1","section":"bucket.instance","name":"tkscyf-8:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.8","timestamp":"2021-02-11T09:47:31.082324615Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZgPd4GGXu--UcP5ytH49TD4","ver":1},"status":{"status":"write"}}},{"id":"1_1613036851.087320_69.1","section":"bucket.instance","name":"tkscyf-8:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.8","timestamp":"2021-02-11T09:47:31.087320996Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZgPd4GGXu--UcP5ytH49TD4","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036851.089023_70.1","section":"bucket","name":"tkscyf-8","timestamp":"2021-02-11T09:47:31.089023916Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MfurQNSSeZct4L9MZ9BG8Gr","ver":1},"status":{"status":"write"}}},{"id":"1_1613036851.093557_71.1","section":"bucket","name":"tkscyf-8","timestamp":"2021-02-11T09:47:31.093557563Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MfurQNSSeZct4L9MZ9BG8Gr","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036851.133487_72.1","section":"bucket.instance","name":"tkscyf-9:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.9","timestamp":"2021-02-11T09:47:31.133487717Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gym2dhV8yR1PgdklxT3K2xe","ver":1},"status":{"status":"write"}}},{"id":"1_1613036851.138269_73.1","section":"bucket.instance","name":"tkscyf-9:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.9","timestamp":"2021-02-11T09:47:31.138269287Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gym2dhV8yR1PgdklxT3K2xe","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036851.139876_74.1","section":"bucket","name":"tkscyf-9","timestamp":"2021-02-11T09:47:31.139876187Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_p0bbqr2Y572m56xpGPFsiZ-","ver":1},"status":{"status":"write"}}},{"id":"1_1613036851.144668_75.1","section":"bucket","name":"tkscyf-9","timestamp":"2021-02-11T09:47:31.144668547Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_p0bbqr2Y572m56xpGPFsiZ-","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036863.874762_76.1","section":"bucket","name":"tkscyf-7","timestamp":"2021-02-11T09:47:43.874762601Z","data":{"read_version":{"tag":"_FCpy5wrdCKP58fp9iJu68WQ","ver":1},"write_version":{"tag":"_FCpy5wrdCKP58fp9iJu68WQ","ver":2},"status":{"status":"remove"}}},{"id":"1_1613036863.880950_77.1","section":"bucket","name":"tkscyf-7","timestamp":"2021-02-11T09:47:43.880950937Z","data":{"read_version":{"tag":"_FCpy5wrdCKP58fp9iJu68WQ","ver":1},"write_version":{"tag":"_FCpy5wrdCKP58fp9iJu68WQ","ver":2},"status":{"status":"complete"}}},{"id":"1_1613036863.956967_78.1","section":"bucket","name":"tkscyf-8","timestamp":"2021-02-11T09:47:43.956967900Z","data":{"read_version":{"tag":"_MfurQNSSeZct4L9MZ9BG8Gr","ver":1},"write_version":{"tag":"_MfurQNSSeZct4L9MZ9BG8Gr","ver":2},"status":{"status":"remove"}}},{"id":"1_1613036863.963661_79.1","section":"bucket","name":"tkscyf-8","timestamp":"2021-02-11T09:47:43.963661824Z","data":{"read_version":{"tag":"_MfurQNSSeZct4L9MZ9BG8Gr","ver":1},"write_version":{"tag":"_MfurQNSSeZct4L9MZ9BG8Gr","ver":2},"status":{"status":"complete"}}},{"id":"1_1613036864.074054_80.1","section":"bucket","name":"tkscyf-9","timestamp":"2021-02-11T09:47:44.074054305Z","data":{"read_version":{"tag":"_p0bbqr2Y572m56xpGPFsiZ-","ver":1},"write_version":{"tag":"_p0bbqr2Y572m56xpGPFsiZ-","ver":2},"status":{"status":"remove"}}},{"id":"1_1613036864.081584_81.1","section":"bucket","name":"tkscyf-9","timestamp":"2021-02-11T09:47:44.081584235Z","data":{"read_version":{"tag":"_p0bbqr2Y572m56xpGPFsiZ-","ver":1},"write_version":{"tag":"_p0bbqr2Y572m56xpGPFsiZ-","ver":2},"status":{"status":"complete"}}}] 2021-02-11T09:53:14.482 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:14.482 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:14.483 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:14.521 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:14.521 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:14.765 INFO:teuthology.orchestra.run.smithi098.stdout:[{"id":"1_1613036708.088554_48.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2021-02-11T09:45:08.088554566Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RA3OvQQVI2OcioTs6vG61ts","ver":1},"status":{"status":"write"}}},{"id":"1_1613036708.096437_52.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2021-02-11T09:45:08.096437057Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RA3OvQQVI2OcioTs6vG61ts","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036865.186535_60.1","section":"bucket.instance","name":"tkscyf-10:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.10","timestamp":"2021-02-11T09:47:45.186535085Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BOQUev-Msz6Gg8HQx3I-Uw_","ver":1},"status":{"status":"write"}}},{"id":"1_1613036865.191608_62.1","section":"bucket.instance","name":"tkscyf-10:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.10","timestamp":"2021-02-11T09:47:45.191608247Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BOQUev-Msz6Gg8HQx3I-Uw_","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036865.193557_63.1","section":"bucket","name":"tkscyf-10","timestamp":"2021-02-11T09:47:45.193557933Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fhFnaBfHBt4OCc3A0Cy3U0l","ver":1},"status":{"status":"write"}}},{"id":"1_1613036865.198089_64.1","section":"bucket","name":"tkscyf-10","timestamp":"2021-02-11T09:47:45.198089937Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fhFnaBfHBt4OCc3A0Cy3U0l","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036865.217116_66.1","section":"bucket.instance","name":"tkscyf-11:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.11","timestamp":"2021-02-11T09:47:45.217116956Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_x_UhJIyltXoCF5sGbvGrvWs","ver":1},"status":{"status":"write"}}},{"id":"1_1613036865.222221_67.1","section":"bucket.instance","name":"tkscyf-11:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.11","timestamp":"2021-02-11T09:47:45.222221108Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_x_UhJIyltXoCF5sGbvGrvWs","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036865.224285_68.1","section":"bucket","name":"tkscyf-11","timestamp":"2021-02-11T09:47:45.224285083Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FRR9uzZkCcSEP86_Kp419wb","ver":1},"status":{"status":"write"}}},{"id":"1_1613036865.228963_69.1","section":"bucket","name":"tkscyf-11","timestamp":"2021-02-11T09:47:45.228963299Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FRR9uzZkCcSEP86_Kp419wb","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036865.267357_70.1","section":"bucket.instance","name":"tkscyf-12:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.12","timestamp":"2021-02-11T09:47:45.267357171Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QtLjmo_4M7YVwSusOVuvKW7","ver":1},"status":{"status":"write"}}},{"id":"1_1613036865.272260_71.1","section":"bucket.instance","name":"tkscyf-12:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.12","timestamp":"2021-02-11T09:47:45.272260535Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QtLjmo_4M7YVwSusOVuvKW7","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036865.273997_72.1","section":"bucket","name":"tkscyf-12","timestamp":"2021-02-11T09:47:45.273997074Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9ZVFThqlMn2Lww6c1ZLx1-O","ver":1},"status":{"status":"write"}}},{"id":"1_1613036865.278468_73.1","section":"bucket","name":"tkscyf-12","timestamp":"2021-02-11T09:47:45.278468126Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9ZVFThqlMn2Lww6c1ZLx1-O","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036885.857139_178.1","section":"bucket.instance","name":"tkscyf-13:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.13","timestamp":"2021-02-11T09:48:05.857139297Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LjOnUQz6uMNf3yxGrmeonuy","ver":1},"status":{"status":"write"}}},{"id":"1_1613036885.861989_179.1","section":"bucket.instance","name":"tkscyf-13:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.13","timestamp":"2021-02-11T09:48:05.861989993Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LjOnUQz6uMNf3yxGrmeonuy","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036885.863884_180.1","section":"bucket","name":"tkscyf-13","timestamp":"2021-02-11T09:48:05.863884773Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f0vlqtQxcyVJgs3vWIVlMY5","ver":1},"status":{"status":"write"}}},{"id":"1_1613036885.868524_181.1","section":"bucket","name":"tkscyf-13","timestamp":"2021-02-11T09:48:05.868524702Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f0vlqtQxcyVJgs3vWIVlMY5","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036885.887239_182.1","section":"bucket.instance","name":"tkscyf-14:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.14","timestamp":"2021-02-11T09:48:05.887239288Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Cb1zieCDGGuK7mRxM9X7cOY","ver":1},"status":{"status":"write"}}},{"id":"1_1613036885.892744_183.1","section":"bucket.instance","name":"tkscyf-14:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.14","timestamp":"2021-02-11T09:48:05.892744502Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Cb1zieCDGGuK7mRxM9X7cOY","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036885.894666_184.1","section":"bucket","name":"tkscyf-14","timestamp":"2021-02-11T09:48:05.894666449Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0d8lwgEigMOkq-FSumkqW71","ver":1},"status":{"status":"write"}}},{"id":"1_1613036885.900682_185.1","section":"bucket","name":"tkscyf-14","timestamp":"2021-02-11T09:48:05.900682555Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0d8lwgEigMOkq-FSumkqW71","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036885.942706_186.1","section":"bucket.instance","name":"tkscyf-15:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.15","timestamp":"2021-02-11T09:48:05.942706611Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-hfQZ5lWiy4o6NP7OEZNQ0h","ver":1},"status":{"status":"write"}}},{"id":"1_1613036885.947464_187.1","section":"bucket.instance","name":"tkscyf-15:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.15","timestamp":"2021-02-11T09:48:05.947464351Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-hfQZ5lWiy4o6NP7OEZNQ0h","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036885.949248_188.1","section":"bucket","name":"tkscyf-15","timestamp":"2021-02-11T09:48:05.949248580Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4gdcmHzvr3iqrlpLrZQpyg1","ver":1},"status":{"status":"write"}}},{"id":"1_1613036885.953989_189.1","section":"bucket","name":"tkscyf-15","timestamp":"2021-02-11T09:48:05.953989244Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4gdcmHzvr3iqrlpLrZQpyg1","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036899.854288_194.1","section":"bucket.instance","name":"tkscyf-16:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.16","timestamp":"2021-02-11T09:48:19.854288478Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BjHGVIhMvUslBcZQfe5MTso","ver":1},"status":{"status":"write"}}},{"id":"1_1613036899.859377_195.1","section":"bucket.instance","name":"tkscyf-16:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.16","timestamp":"2021-02-11T09:48:19.859377386Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BjHGVIhMvUslBcZQfe5MTso","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036899.861235_196.1","section":"bucket","name":"tkscyf-16","timestamp":"2021-02-11T09:48:19.861235489Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_mBUX065ZmzTTf0GD1alWPkY","ver":1},"status":{"status":"write"}}},{"id":"1_1613036899.866335_197.1","section":"bucket","name":"tkscyf-16","timestamp":"2021-02-11T09:48:19.866335427Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_mBUX065ZmzTTf0GD1alWPkY","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036899.888709_198.1","section":"bucket.instance","name":"tkscyf-17:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.17","timestamp":"2021-02-11T09:48:19.888709670Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_upQr8yx99-au6_AWkKxYuHu","ver":1},"status":{"status":"write"}}},{"id":"1_1613036899.894272_199.1","section":"bucket.instance","name":"tkscyf-17:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.17","timestamp":"2021-02-11T09:48:19.894272943Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_upQr8yx99-au6_AWkKxYuHu","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036899.896430_200.1","section":"bucket","name":"tkscyf-17","timestamp":"2021-02-11T09:48:19.896430934Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_EVefVFc5TujdFr9n6vuddup","ver":1},"status":{"status":"write"}}},{"id":"1_1613036899.901399_201.1","section":"bucket","name":"tkscyf-17","timestamp":"2021-02-11T09:48:19.901399970Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_EVefVFc5TujdFr9n6vuddup","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036899.942394_202.1","section":"bucket.instance","name":"tkscyf-18:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.18","timestamp":"2021-02-11T09:48:19.942394398Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eLelEHOSgz_5RWVJgejlcwE","ver":1},"status":{"status":"write"}}},{"id":"1_1613036899.947586_203.1","section":"bucket.instance","name":"tkscyf-18:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.18","timestamp":"2021-02-11T09:48:19.947586263Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eLelEHOSgz_5RWVJgejlcwE","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036899.949797_204.1","section":"bucket","name":"tkscyf-18","timestamp":"2021-02-11T09:48:19.949797527Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nnXNYQhXpslLZMkcMsbaLYW","ver":1},"status":{"status":"write"}}},{"id":"1_1613036899.954577_205.1","section":"bucket","name":"tkscyf-18","timestamp":"2021-02-11T09:48:19.954577937Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nnXNYQhXpslLZMkcMsbaLYW","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036899.976313_206.1","section":"bucket.instance","name":"tkscyf-16:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.16","timestamp":"2021-02-11T09:48:19.976313350Z","data":{"read_version":{"tag":"_BjHGVIhMvUslBcZQfe5MTso","ver":1},"write_version":{"tag":"_BjHGVIhMvUslBcZQfe5MTso","ver":2},"status":{"status":"write"}}},{"id":"1_1613036899.981307_207.1","section":"bucket.instance","name":"tkscyf-16:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.16","timestamp":"2021-02-11T09:48:19.981307840Z","data":{"read_version":{"tag":"_BjHGVIhMvUslBcZQfe5MTso","ver":1},"write_version":{"tag":"_BjHGVIhMvUslBcZQfe5MTso","ver":2},"status":{"status":"complete"}}},{"id":"1_1613036899.986228_208.1","section":"bucket.instance","name":"tkscyf-17:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.17","timestamp":"2021-02-11T09:48:19.986228789Z","data":{"read_version":{"tag":"_upQr8yx99-au6_AWkKxYuHu","ver":1},"write_version":{"tag":"_upQr8yx99-au6_AWkKxYuHu","ver":2},"status":{"status":"write"}}},{"id":"1_1613036899.991766_209.1","section":"bucket.instance","name":"tkscyf-17:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.17","timestamp":"2021-02-11T09:48:19.991766410Z","data":{"read_version":{"tag":"_upQr8yx99-au6_AWkKxYuHu","ver":1},"write_version":{"tag":"_upQr8yx99-au6_AWkKxYuHu","ver":2},"status":{"status":"complete"}}},{"id":"1_1613036900.005531_210.1","section":"bucket.instance","name":"tkscyf-18:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.18","timestamp":"2021-02-11T09:48:20.005531886Z","data":{"read_version":{"tag":"_eLelEHOSgz_5RWVJgejlcwE","ver":1},"write_version":{"tag":"_eLelEHOSgz_5RWVJgejlcwE","ver":2},"status":{"status":"write"}}},{"id":"1_1613036900.015316_211.1","section":"bucket.instance","name":"tkscyf-18:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.18","timestamp":"2021-02-11T09:48:20.015316565Z","data":{"read_version":{"tag":"_eLelEHOSgz_5RWVJgejlcwE","ver":1},"write_version":{"tag":"_eLelEHOSgz_5RWVJgejlcwE","ver":2},"status":{"status":"complete"}}},{"id":"1_1613036950.507710_227.1","section":"bucket.instance","name":"tkscyf-19:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.19","timestamp":"2021-02-11T09:49:10.507710607Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_G8JIgZ6gi-tWTxNuGHdxLfV","ver":1},"status":{"status":"write"}}},{"id":"1_1613036950.512728_228.1","section":"bucket.instance","name":"tkscyf-19:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.19","timestamp":"2021-02-11T09:49:10.512728298Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_G8JIgZ6gi-tWTxNuGHdxLfV","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036950.514564_229.1","section":"bucket","name":"tkscyf-19","timestamp":"2021-02-11T09:49:10.514564712Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rUH9m8tDhvujVPRfyQMubOa","ver":1},"status":{"status":"write"}}},{"id":"1_1613036950.519470_230.1","section":"bucket","name":"tkscyf-19","timestamp":"2021-02-11T09:49:10.519470166Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rUH9m8tDhvujVPRfyQMubOa","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036950.525237_231.1","section":"bucket.instance","name":"tkscyf-19:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.19","timestamp":"2021-02-11T09:49:10.525237169Z","data":{"read_version":{"tag":"_G8JIgZ6gi-tWTxNuGHdxLfV","ver":1},"write_version":{"tag":"_G8JIgZ6gi-tWTxNuGHdxLfV","ver":2},"status":{"status":"write"}}},{"id":"1_1613036950.530761_232.1","section":"bucket.instance","name":"tkscyf-19:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.19","timestamp":"2021-02-11T09:49:10.530761661Z","data":{"read_version":{"tag":"_G8JIgZ6gi-tWTxNuGHdxLfV","ver":1},"write_version":{"tag":"_G8JIgZ6gi-tWTxNuGHdxLfV","ver":2},"status":{"status":"complete"}}},{"id":"1_1613036998.267812_238.1","section":"bucket.instance","name":"tkscyf-20:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.20","timestamp":"2021-02-11T09:49:58.267812499Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3X25hzqCxgrngeWBjXAq3MW","ver":1},"status":{"status":"write"}}},{"id":"1_1613036998.273260_239.1","section":"bucket.instance","name":"tkscyf-20:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.20","timestamp":"2021-02-11T09:49:58.273260665Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3X25hzqCxgrngeWBjXAq3MW","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036998.275321_240.1","section":"bucket","name":"tkscyf-20","timestamp":"2021-02-11T09:49:58.275321795Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GSd-2J0lHNWhUfK9gtbzsaz","ver":1},"status":{"status":"write"}}},{"id":"1_1613036998.280111_241.1","section":"bucket","name":"tkscyf-20","timestamp":"2021-02-11T09:49:58.280111713Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GSd-2J0lHNWhUfK9gtbzsaz","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037007.140764_243.1","section":"bucket.instance","name":"tkscyf-20:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.20","timestamp":"2021-02-11T09:50:07.140764026Z","data":{"read_version":{"tag":"_3X25hzqCxgrngeWBjXAq3MW","ver":1},"write_version":{"tag":"_3X25hzqCxgrngeWBjXAq3MW","ver":2},"status":{"status":"write"}}},{"id":"1_1613037007.150589_244.1","section":"bucket.instance","name":"tkscyf-20:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.20","timestamp":"2021-02-11T09:50:07.150589879Z","data":{"read_version":{"tag":"_3X25hzqCxgrngeWBjXAq3MW","ver":1},"write_version":{"tag":"_3X25hzqCxgrngeWBjXAq3MW","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037012.304895_245.1","section":"bucket.instance","name":"tkscyf-20:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.20","timestamp":"2021-02-11T09:50:12.304895189Z","data":{"read_version":{"tag":"_3X25hzqCxgrngeWBjXAq3MW","ver":2},"write_version":{"tag":"_3X25hzqCxgrngeWBjXAq3MW","ver":3},"status":{"status":"write"}}},{"id":"1_1613037012.316265_246.1","section":"bucket.instance","name":"tkscyf-20:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.20","timestamp":"2021-02-11T09:50:12.316265099Z","data":{"read_version":{"tag":"_3X25hzqCxgrngeWBjXAq3MW","ver":2},"write_version":{"tag":"_3X25hzqCxgrngeWBjXAq3MW","ver":3},"status":{"status":"complete"}}},{"id":"1_1613037016.767451_247.1","section":"bucket.instance","name":"tkscyf-21:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.21","timestamp":"2021-02-11T09:50:16.767451636Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_usNhex5LVJs2K5fhCTS-E9B","ver":1},"status":{"status":"write"}}},{"id":"1_1613037016.772791_248.1","section":"bucket.instance","name":"tkscyf-21:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.21","timestamp":"2021-02-11T09:50:16.772791796Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_usNhex5LVJs2K5fhCTS-E9B","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037016.774734_249.1","section":"bucket","name":"tkscyf-21","timestamp":"2021-02-11T09:50:16.774734911Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_oS8fsXNPnQ3YSGsw6yEPI6c","ver":1},"status":{"status":"write"}}},{"id":"1_1613037016.779555_250.1","section":"bucket","name":"tkscyf-21","timestamp":"2021-02-11T09:50:16.779555063Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_oS8fsXNPnQ3YSGsw6yEPI6c","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037016.800629_251.1","section":"bucket.instance","name":"tkscyf-22:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.22","timestamp":"2021-02-11T09:50:16.800629460Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Q5XnU57eMsSQj5AxurjrM1s","ver":1},"status":{"status":"write"}}},{"id":"1_1613037016.807098_252.1","section":"bucket.instance","name":"tkscyf-22:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.22","timestamp":"2021-02-11T09:50:16.807098120Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Q5XnU57eMsSQj5AxurjrM1s","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037016.809071_253.1","section":"bucket","name":"tkscyf-22","timestamp":"2021-02-11T09:50:16.809071480Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Jcw-M7gbUwOe2t495PGK9vo","ver":1},"status":{"status":"write"}}},{"id":"1_1613037016.813672_254.1","section":"bucket","name":"tkscyf-22","timestamp":"2021-02-11T09:50:16.813672741Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Jcw-M7gbUwOe2t495PGK9vo","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037016.860072_255.1","section":"bucket.instance","name":"tkscyf-23:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.23","timestamp":"2021-02-11T09:50:16.860072689Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GCLr8VFsBOKFg5ZuZUx_GvZ","ver":1},"status":{"status":"write"}}},{"id":"1_1613037016.866566_256.1","section":"bucket.instance","name":"tkscyf-23:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.23","timestamp":"2021-02-11T09:50:16.866566184Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GCLr8VFsBOKFg5ZuZUx_GvZ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037016.869661_257.1","section":"bucket","name":"tkscyf-23","timestamp":"2021-02-11T09:50:16.869661986Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8HFWSnbWVi7UHPWj_xF4vXf","ver":1},"status":{"status":"write"}}},{"id":"1_1613037016.879001_258.1","section":"bucket","name":"tkscyf-23","timestamp":"2021-02-11T09:50:16.879001731Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8HFWSnbWVi7UHPWj_xF4vXf","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037016.897902_259.1","section":"bucket.instance","name":"tkscyf-21:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.21","timestamp":"2021-02-11T09:50:16.897902097Z","data":{"read_version":{"tag":"_usNhex5LVJs2K5fhCTS-E9B","ver":1},"write_version":{"tag":"_usNhex5LVJs2K5fhCTS-E9B","ver":2},"status":{"status":"write"}}},{"id":"1_1613037016.902295_260.1","section":"bucket.instance","name":"tkscyf-21:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.21","timestamp":"2021-02-11T09:50:16.902295553Z","data":{"read_version":{"tag":"_usNhex5LVJs2K5fhCTS-E9B","ver":1},"write_version":{"tag":"_usNhex5LVJs2K5fhCTS-E9B","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037016.906251_261.1","section":"bucket.instance","name":"tkscyf-22:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.22","timestamp":"2021-02-11T09:50:16.906251476Z","data":{"read_version":{"tag":"_Q5XnU57eMsSQj5AxurjrM1s","ver":1},"write_version":{"tag":"_Q5XnU57eMsSQj5AxurjrM1s","ver":2},"status":{"status":"write"}}},{"id":"1_1613037016.911598_262.1","section":"bucket.instance","name":"tkscyf-22:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.22","timestamp":"2021-02-11T09:50:16.911598777Z","data":{"read_version":{"tag":"_Q5XnU57eMsSQj5AxurjrM1s","ver":1},"write_version":{"tag":"_Q5XnU57eMsSQj5AxurjrM1s","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037016.926597_263.1","section":"bucket.instance","name":"tkscyf-23:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.23","timestamp":"2021-02-11T09:50:16.926597890Z","data":{"read_version":{"tag":"_GCLr8VFsBOKFg5ZuZUx_GvZ","ver":1},"write_version":{"tag":"_GCLr8VFsBOKFg5ZuZUx_GvZ","ver":2},"status":{"status":"write"}}},{"id":"1_1613037016.931312_264.1","section":"bucket.instance","name":"tkscyf-23:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.23","timestamp":"2021-02-11T09:50:16.931312694Z","data":{"read_version":{"tag":"_GCLr8VFsBOKFg5ZuZUx_GvZ","ver":1},"write_version":{"tag":"_GCLr8VFsBOKFg5ZuZUx_GvZ","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037035.238424_265.1","section":"bucket.instance","name":"tkscyf-24:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.24","timestamp":"2021-02-11T09:50:35.238424865Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bb0kUiSwNRO-U3TZ4E1z9m5","ver":1},"status":{"status":"write"}}},{"id":"1_1613037035.243640_266.1","section":"bucket.instance","name":"tkscyf-24:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.24","timestamp":"2021-02-11T09:50:35.243640241Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bb0kUiSwNRO-U3TZ4E1z9m5","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037035.245475_267.1","section":"bucket","name":"tkscyf-24","timestamp":"2021-02-11T09:50:35.245475848Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7Yg_eLnx93F7TXgKcKQAxT5","ver":1},"status":{"status":"write"}}},{"id":"1_1613037035.250290_268.1","section":"bucket","name":"tkscyf-24","timestamp":"2021-02-11T09:50:35.250290672Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7Yg_eLnx93F7TXgKcKQAxT5","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037035.274261_269.1","section":"bucket.instance","name":"tkscyf-25:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.25","timestamp":"2021-02-11T09:50:35.274261140Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1nnotJE0A2l-djRVSBIWs9E","ver":1},"status":{"status":"write"}}},{"id":"1_1613037035.281072_270.1","section":"bucket.instance","name":"tkscyf-25:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.25","timestamp":"2021-02-11T09:50:35.281072011Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1nnotJE0A2l-djRVSBIWs9E","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037035.283085_271.1","section":"bucket","name":"tkscyf-25","timestamp":"2021-02-11T09:50:35.283085981Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PBWPxzfNM3rpuuSRbw9WWJ4","ver":1},"status":{"status":"write"}}},{"id":"1_1613037035.287331_272.1","section":"bucket","name":"tkscyf-25","timestamp":"2021-02-11T09:50:35.287331936Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PBWPxzfNM3rpuuSRbw9WWJ4","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037035.327494_273.1","section":"bucket.instance","name":"tkscyf-26:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.26","timestamp":"2021-02-11T09:50:35.327494695Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yBW2OnCxamt6jwtfytnSIYp","ver":1},"status":{"status":"write"}}},{"id":"1_1613037035.336740_274.1","section":"bucket.instance","name":"tkscyf-26:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.26","timestamp":"2021-02-11T09:50:35.336740442Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yBW2OnCxamt6jwtfytnSIYp","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037035.338820_275.1","section":"bucket","name":"tkscyf-26","timestamp":"2021-02-11T09:50:35.338820669Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GBS2JL6Meg0eeenzd3LFcRc","ver":1},"status":{"status":"write"}}},{"id":"1_1613037035.343318_276.1","section":"bucket","name":"tkscyf-26","timestamp":"2021-02-11T09:50:35.343318410Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GBS2JL6Meg0eeenzd3LFcRc","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037035.361989_277.1","section":"bucket.instance","name":"tkscyf-24:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.24","timestamp":"2021-02-11T09:50:35.361989484Z","data":{"read_version":{"tag":"_bb0kUiSwNRO-U3TZ4E1z9m5","ver":1},"write_version":{"tag":"_bb0kUiSwNRO-U3TZ4E1z9m5","ver":2},"status":{"status":"write"}}},{"id":"1_1613037035.367759_278.1","section":"bucket.instance","name":"tkscyf-24:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.24","timestamp":"2021-02-11T09:50:35.367759053Z","data":{"read_version":{"tag":"_bb0kUiSwNRO-U3TZ4E1z9m5","ver":1},"write_version":{"tag":"_bb0kUiSwNRO-U3TZ4E1z9m5","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037035.371457_279.1","section":"bucket.instance","name":"tkscyf-24:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.24","timestamp":"2021-02-11T09:50:35.371457040Z","data":{"read_version":{"tag":"_bb0kUiSwNRO-U3TZ4E1z9m5","ver":2},"write_version":{"tag":"_bb0kUiSwNRO-U3TZ4E1z9m5","ver":3},"status":{"status":"write"}}},{"id":"1_1613037035.376968_280.1","section":"bucket.instance","name":"tkscyf-24:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.24","timestamp":"2021-02-11T09:50:35.376968483Z","data":{"read_version":{"tag":"_bb0kUiSwNRO-U3TZ4E1z9m5","ver":2},"write_version":{"tag":"_bb0kUiSwNRO-U3TZ4E1z9m5","ver":3},"status":{"status":"complete"}}},{"id":"1_1613037035.382067_281.1","section":"bucket.instance","name":"tkscyf-25:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.25","timestamp":"2021-02-11T09:50:35.382067509Z","data":{"read_version":{"tag":"_1nnotJE0A2l-djRVSBIWs9E","ver":1},"write_version":{"tag":"_1nnotJE0A2l-djRVSBIWs9E","ver":2},"status":{"status":"write"}}},{"id":"1_1613037035.388230_282.1","section":"bucket.instance","name":"tkscyf-25:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.25","timestamp":"2021-02-11T09:50:35.388230116Z","data":{"read_version":{"tag":"_1nnotJE0A2l-djRVSBIWs9E","ver":1},"write_version":{"tag":"_1nnotJE0A2l-djRVSBIWs9E","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037035.399024_283.1","section":"bucket.instance","name":"tkscyf-25:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.25","timestamp":"2021-02-11T09:50:35.399024419Z","data":{"read_version":{"tag":"_1nnotJE0A2l-djRVSBIWs9E","ver":2},"write_version":{"tag":"_1nnotJE0A2l-djRVSBIWs9E","ver":3},"status":{"status":"write"}}},{"id":"1_1613037035.406913_284.1","section":"bucket.instance","name":"tkscyf-25:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.25","timestamp":"2021-02-11T09:50:35.406913878Z","data":{"read_version":{"tag":"_1nnotJE0A2l-djRVSBIWs9E","ver":2},"write_version":{"tag":"_1nnotJE0A2l-djRVSBIWs9E","ver":3},"status":{"status":"complete"}}},{"id":"1_1613037035.418911_285.1","section":"bucket.instance","name":"tkscyf-26:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.26","timestamp":"2021-02-11T09:50:35.418911493Z","data":{"read_version":{"tag":"_yBW2OnCxamt6jwtfytnSIYp","ver":1},"write_version":{"tag":"_yBW2OnCxamt6jwtfytnSIYp","ver":2},"status":{"status":"write"}}},{"id":"1_1613037035.427825_286.1","section":"bucket.instance","name":"tkscyf-26:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.26","timestamp":"2021-02-11T09:50:35.427825538Z","data":{"read_version":{"tag":"_yBW2OnCxamt6jwtfytnSIYp","ver":1},"write_version":{"tag":"_yBW2OnCxamt6jwtfytnSIYp","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037035.439773_287.1","section":"bucket.instance","name":"tkscyf-26:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.26","timestamp":"2021-02-11T09:50:35.439773122Z","data":{"read_version":{"tag":"_yBW2OnCxamt6jwtfytnSIYp","ver":2},"write_version":{"tag":"_yBW2OnCxamt6jwtfytnSIYp","ver":3},"status":{"status":"write"}}},{"id":"1_1613037035.445782_288.1","section":"bucket.instance","name":"tkscyf-26:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.26","timestamp":"2021-02-11T09:50:35.445782025Z","data":{"read_version":{"tag":"_yBW2OnCxamt6jwtfytnSIYp","ver":2},"write_version":{"tag":"_yBW2OnCxamt6jwtfytnSIYp","ver":3},"status":{"status":"complete"}}},{"id":"1_1613037049.014616_289.1","section":"bucket.instance","name":"tkscyf-27:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.27","timestamp":"2021-02-11T09:50:49.014616484Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HPEAummOtXzD3iTYGsjL78L","ver":1},"status":{"status":"write"}}},{"id":"1_1613037049.020486_290.1","section":"bucket.instance","name":"tkscyf-27:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.27","timestamp":"2021-02-11T09:50:49.020486409Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HPEAummOtXzD3iTYGsjL78L","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037049.022446_291.1","section":"bucket","name":"tkscyf-27","timestamp":"2021-02-11T09:50:49.022446056Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HzrXfJNvB4DsZqVjRjsmkDL","ver":1},"status":{"status":"write"}}},{"id":"1_1613037049.026934_292.1","section":"bucket","name":"tkscyf-27","timestamp":"2021-02-11T09:50:49.026934689Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HzrXfJNvB4DsZqVjRjsmkDL","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037049.052544_293.1","section":"bucket.instance","name":"tkscyf-28:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.28","timestamp":"2021-02-11T09:50:49.052544751Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-VSdKrlRKUfuwloTfMI6Xuv","ver":1},"status":{"status":"write"}}},{"id":"1_1613037049.057739_294.1","section":"bucket.instance","name":"tkscyf-28:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.28","timestamp":"2021-02-11T09:50:49.057739941Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-VSdKrlRKUfuwloTfMI6Xuv","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037049.059973_295.1","section":"bucket","name":"tkscyf-28","timestamp":"2021-02-11T09:50:49.059973620Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-BX9bB1FIeOQc9AK3EsxPMy","ver":1},"status":{"status":"write"}}},{"id":"1_1613037049.064978_296.1","section":"bucket","name":"tkscyf-28","timestamp":"2021-02-11T09:50:49.064978209Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-BX9bB1FIeOQc9AK3EsxPMy","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037049.129163_297.1","section":"bucket.instance","name":"tkscyf-29:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.29","timestamp":"2021-02-11T09:50:49.129163449Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hyJCgZ3ZqhjZ2B82NLQtbFj","ver":1},"status":{"status":"write"}}},{"id":"1_1613037049.139199_298.1","section":"bucket.instance","name":"tkscyf-29:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.29","timestamp":"2021-02-11T09:50:49.139199855Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hyJCgZ3ZqhjZ2B82NLQtbFj","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037049.141005_299.1","section":"bucket","name":"tkscyf-29","timestamp":"2021-02-11T09:50:49.141005489Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1eRc33iNyGdkXUg6pUKi_qn","ver":1},"status":{"status":"write"}}},{"id":"1_1613037049.145434_300.1","section":"bucket","name":"tkscyf-29","timestamp":"2021-02-11T09:50:49.145434314Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1eRc33iNyGdkXUg6pUKi_qn","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037049.166096_301.1","section":"bucket.instance","name":"tkscyf-27:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.27","timestamp":"2021-02-11T09:50:49.166096762Z","data":{"read_version":{"tag":"_HPEAummOtXzD3iTYGsjL78L","ver":1},"write_version":{"tag":"_HPEAummOtXzD3iTYGsjL78L","ver":2},"status":{"status":"write"}}},{"id":"1_1613037049.171244_302.1","section":"bucket.instance","name":"tkscyf-27:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.27","timestamp":"2021-02-11T09:50:49.171244329Z","data":{"read_version":{"tag":"_HPEAummOtXzD3iTYGsjL78L","ver":1},"write_version":{"tag":"_HPEAummOtXzD3iTYGsjL78L","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037049.219869_303.1","section":"bucket.instance","name":"tkscyf-28:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.28","timestamp":"2021-02-11T09:50:49.219869180Z","data":{"read_version":{"tag":"_-VSdKrlRKUfuwloTfMI6Xuv","ver":1},"write_version":{"tag":"_-VSdKrlRKUfuwloTfMI6Xuv","ver":2},"status":{"status":"write"}}},{"id":"1_1613037049.230173_304.1","section":"bucket.instance","name":"tkscyf-28:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.28","timestamp":"2021-02-11T09:50:49.230173871Z","data":{"read_version":{"tag":"_-VSdKrlRKUfuwloTfMI6Xuv","ver":1},"write_version":{"tag":"_-VSdKrlRKUfuwloTfMI6Xuv","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037049.287108_305.1","section":"bucket.instance","name":"tkscyf-29:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.29","timestamp":"2021-02-11T09:50:49.287108951Z","data":{"read_version":{"tag":"_hyJCgZ3ZqhjZ2B82NLQtbFj","ver":1},"write_version":{"tag":"_hyJCgZ3ZqhjZ2B82NLQtbFj","ver":2},"status":{"status":"write"}}},{"id":"1_1613037049.300847_306.1","section":"bucket.instance","name":"tkscyf-29:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.29","timestamp":"2021-02-11T09:50:49.300847185Z","data":{"read_version":{"tag":"_hyJCgZ3ZqhjZ2B82NLQtbFj","ver":1},"write_version":{"tag":"_hyJCgZ3ZqhjZ2B82NLQtbFj","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037049.364299_307.1","section":"bucket.instance","name":"tkscyf-30:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.30","timestamp":"2021-02-11T09:50:49.364299749Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Es0JugEkT-f3o-riZAdqnrS","ver":1},"status":{"status":"write"}}},{"id":"1_1613037049.370623_308.1","section":"bucket.instance","name":"tkscyf-30:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.30","timestamp":"2021-02-11T09:50:49.370623916Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Es0JugEkT-f3o-riZAdqnrS","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037049.372900_309.1","section":"bucket","name":"tkscyf-30","timestamp":"2021-02-11T09:50:49.372900977Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0ipPDFF-oXK1PAXV9ImeRz8","ver":1},"status":{"status":"write"}}},{"id":"1_1613037049.377959_310.1","section":"bucket","name":"tkscyf-30","timestamp":"2021-02-11T09:50:49.377959725Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0ipPDFF-oXK1PAXV9ImeRz8","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037049.400272_311.1","section":"bucket.instance","name":"tkscyf-31:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.31","timestamp":"2021-02-11T09:50:49.400272843Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_J_JeLJIs-yKAxn3I-jH6tun","ver":1},"status":{"status":"write"}}},{"id":"1_1613037049.406204_312.1","section":"bucket.instance","name":"tkscyf-31:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.31","timestamp":"2021-02-11T09:50:49.406204831Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_J_JeLJIs-yKAxn3I-jH6tun","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037049.407984_313.1","section":"bucket","name":"tkscyf-31","timestamp":"2021-02-11T09:50:49.407984579Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AtSnm2w_MDdPl4ZTuW8BBX1","ver":1},"status":{"status":"write"}}},{"id":"1_1613037049.412626_314.1","section":"bucket","name":"tkscyf-31","timestamp":"2021-02-11T09:50:49.412626629Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AtSnm2w_MDdPl4ZTuW8BBX1","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037049.456098_315.1","section":"bucket.instance","name":"tkscyf-32:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.32","timestamp":"2021-02-11T09:50:49.456098996Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jg7YdCRiL4_wX7QgQ9ZEBCK","ver":1},"status":{"status":"write"}}},{"id":"1_1613037049.461384_316.1","section":"bucket.instance","name":"tkscyf-32:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.32","timestamp":"2021-02-11T09:50:49.461384431Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jg7YdCRiL4_wX7QgQ9ZEBCK","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037049.463339_317.1","section":"bucket","name":"tkscyf-32","timestamp":"2021-02-11T09:50:49.463339008Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eg1axhWTHrzA2NWrcdnReCD","ver":1},"status":{"status":"write"}}},{"id":"1_1613037049.468298_318.1","section":"bucket","name":"tkscyf-32","timestamp":"2021-02-11T09:50:49.468298585Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eg1axhWTHrzA2NWrcdnReCD","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037049.493719_319.1","section":"bucket.instance","name":"tkscyf-30:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.30","timestamp":"2021-02-11T09:50:49.493719070Z","data":{"read_version":{"tag":"_Es0JugEkT-f3o-riZAdqnrS","ver":1},"write_version":{"tag":"_Es0JugEkT-f3o-riZAdqnrS","ver":2},"status":{"status":"write"}}},{"id":"1_1613037049.498976_320.1","section":"bucket.instance","name":"tkscyf-30:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.30","timestamp":"2021-02-11T09:50:49.498976233Z","data":{"read_version":{"tag":"_Es0JugEkT-f3o-riZAdqnrS","ver":1},"write_version":{"tag":"_Es0JugEkT-f3o-riZAdqnrS","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037049.547179_321.1","section":"bucket.instance","name":"tkscyf-31:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.31","timestamp":"2021-02-11T09:50:49.547179938Z","data":{"read_version":{"tag":"_J_JeLJIs-yKAxn3I-jH6tun","ver":1},"write_version":{"tag":"_J_JeLJIs-yKAxn3I-jH6tun","ver":2},"status":{"status":"write"}}},{"id":"1_1613037049.552155_322.1","section":"bucket.instance","name":"tkscyf-31:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.31","timestamp":"2021-02-11T09:50:49.552155032Z","data":{"read_version":{"tag":"_J_JeLJIs-yKAxn3I-jH6tun","ver":1},"write_version":{"tag":"_J_JeLJIs-yKAxn3I-jH6tun","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037049.603090_323.1","section":"bucket.instance","name":"tkscyf-32:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.32","timestamp":"2021-02-11T09:50:49.603090607Z","data":{"read_version":{"tag":"_jg7YdCRiL4_wX7QgQ9ZEBCK","ver":1},"write_version":{"tag":"_jg7YdCRiL4_wX7QgQ9ZEBCK","ver":2},"status":{"status":"write"}}},{"id":"1_1613037049.608976_324.1","section":"bucket.instance","name":"tkscyf-32:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.32","timestamp":"2021-02-11T09:50:49.608976182Z","data":{"read_version":{"tag":"_jg7YdCRiL4_wX7QgQ9ZEBCK","ver":1},"write_version":{"tag":"_jg7YdCRiL4_wX7QgQ9ZEBCK","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037049.680125_325.1","section":"bucket.instance","name":"tkscyf-33:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.33","timestamp":"2021-02-11T09:50:49.680125618Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eZRAP_8ajxLAWs4hu7nqOwz","ver":1},"status":{"status":"write"}}},{"id":"1_1613037049.687075_326.1","section":"bucket.instance","name":"tkscyf-33:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.33","timestamp":"2021-02-11T09:50:49.687075481Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eZRAP_8ajxLAWs4hu7nqOwz","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037049.692386_327.1","section":"bucket","name":"tkscyf-33","timestamp":"2021-02-11T09:50:49.692386405Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-6dR6mEXl2aMVHaPbOLG9sm","ver":1},"status":{"status":"write"}}},{"id":"1_1613037049.699844_328.1","section":"bucket","name":"tkscyf-33","timestamp":"2021-02-11T09:50:49.699844848Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-6dR6mEXl2aMVHaPbOLG9sm","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037049.725396_329.1","section":"bucket.instance","name":"tkscyf-34:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.34","timestamp":"2021-02-11T09:50:49.725396604Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_k7kM2SM39DqWgu3NfoI4bqF","ver":1},"status":{"status":"write"}}},{"id":"1_1613037049.731707_330.1","section":"bucket.instance","name":"tkscyf-34:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.34","timestamp":"2021-02-11T09:50:49.731707348Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_k7kM2SM39DqWgu3NfoI4bqF","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037049.735696_331.1","section":"bucket","name":"tkscyf-34","timestamp":"2021-02-11T09:50:49.735696647Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_L016eB3Fp2Gc1WBb3ihrD9w","ver":1},"status":{"status":"write"}}},{"id":"1_1613037049.740587_332.1","section":"bucket","name":"tkscyf-34","timestamp":"2021-02-11T09:50:49.740587087Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_L016eB3Fp2Gc1WBb3ihrD9w","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037049.780026_333.1","section":"bucket.instance","name":"tkscyf-35:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.35","timestamp":"2021-02-11T09:50:49.780026803Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KosuwqaTc-yYnhDuNK50-jc","ver":1},"status":{"status":"write"}}},{"id":"1_1613037049.785523_334.1","section":"bucket.instance","name":"tkscyf-35:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.35","timestamp":"2021-02-11T09:50:49.785523728Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KosuwqaTc-yYnhDuNK50-jc","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037049.787540_335.1","section":"bucket","name":"tkscyf-35","timestamp":"2021-02-11T09:50:49.787540619Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wBCuU48Ox_uSnLpuXpSWi95","ver":1},"status":{"status":"write"}}},{"id":"1_1613037049.792438_336.1","section":"bucket","name":"tkscyf-35","timestamp":"2021-02-11T09:50:49.792438114Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wBCuU48Ox_uSnLpuXpSWi95","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037049.814263_337.1","section":"bucket.instance","name":"tkscyf-33:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.33","timestamp":"2021-02-11T09:50:49.814263352Z","data":{"read_version":{"tag":"_eZRAP_8ajxLAWs4hu7nqOwz","ver":1},"write_version":{"tag":"_eZRAP_8ajxLAWs4hu7nqOwz","ver":2},"status":{"status":"write"}}},{"id":"1_1613037049.819262_338.1","section":"bucket.instance","name":"tkscyf-33:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.33","timestamp":"2021-02-11T09:50:49.819262498Z","data":{"read_version":{"tag":"_eZRAP_8ajxLAWs4hu7nqOwz","ver":1},"write_version":{"tag":"_eZRAP_8ajxLAWs4hu7nqOwz","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037050.867169_339.1","section":"bucket.instance","name":"tkscyf-34:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.34","timestamp":"2021-02-11T09:50:50.867169930Z","data":{"read_version":{"tag":"_k7kM2SM39DqWgu3NfoI4bqF","ver":1},"write_version":{"tag":"_k7kM2SM39DqWgu3NfoI4bqF","ver":2},"status":{"status":"write"}}},{"id":"1_1613037050.879211_340.1","section":"bucket.instance","name":"tkscyf-34:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.34","timestamp":"2021-02-11T09:50:50.879211406Z","data":{"read_version":{"tag":"_k7kM2SM39DqWgu3NfoI4bqF","ver":1},"write_version":{"tag":"_k7kM2SM39DqWgu3NfoI4bqF","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037051.935225_341.1","section":"bucket.instance","name":"tkscyf-35:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.35","timestamp":"2021-02-11T09:50:51.935225906Z","data":{"read_version":{"tag":"_KosuwqaTc-yYnhDuNK50-jc","ver":1},"write_version":{"tag":"_KosuwqaTc-yYnhDuNK50-jc","ver":2},"status":{"status":"write"}}},{"id":"1_1613037051.946001_342.1","section":"bucket.instance","name":"tkscyf-35:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.35","timestamp":"2021-02-11T09:50:51.946001361Z","data":{"read_version":{"tag":"_KosuwqaTc-yYnhDuNK50-jc","ver":1},"write_version":{"tag":"_KosuwqaTc-yYnhDuNK50-jc","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037056.396292_343.1","section":"bucket.instance","name":"tkscyf-36:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.36","timestamp":"2021-02-11T09:50:56.396292831Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ssZsgWPzFUlbtHH72rboNt2","ver":1},"status":{"status":"write"}}},{"id":"1_1613037056.402315_344.1","section":"bucket.instance","name":"tkscyf-36:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.36","timestamp":"2021-02-11T09:50:56.402315663Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ssZsgWPzFUlbtHH72rboNt2","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037056.404168_345.1","section":"bucket","name":"tkscyf-36","timestamp":"2021-02-11T09:50:56.404168283Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_R9PAwh1OvxD1UjEPuuj8qio","ver":1},"status":{"status":"write"}}},{"id":"1_1613037056.408811_346.1","section":"bucket","name":"tkscyf-36","timestamp":"2021-02-11T09:50:56.408811301Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_R9PAwh1OvxD1UjEPuuj8qio","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037056.437739_347.1","section":"bucket.instance","name":"tkscyf-37:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.37","timestamp":"2021-02-11T09:50:56.437739913Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HWaNKKBo9OCATJepv1PpQ9b","ver":1},"status":{"status":"write"}}},{"id":"1_1613037056.442573_348.1","section":"bucket.instance","name":"tkscyf-37:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.37","timestamp":"2021-02-11T09:50:56.442573229Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HWaNKKBo9OCATJepv1PpQ9b","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037056.444396_349.1","section":"bucket","name":"tkscyf-37","timestamp":"2021-02-11T09:50:56.444396764Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RbeSoBdgaTvi80yVezMNU9a","ver":1},"status":{"status":"write"}}},{"id":"1_1613037056.449127_350.1","section":"bucket","name":"tkscyf-37","timestamp":"2021-02-11T09:50:56.449127942Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RbeSoBdgaTvi80yVezMNU9a","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037056.503850_351.1","section":"bucket.instance","name":"tkscyf-38:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.38","timestamp":"2021-02-11T09:50:56.503850524Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_M7gYPbaihLb1WY3hhZTdAtg","ver":1},"status":{"status":"write"}}},{"id":"1_1613037056.508532_352.1","section":"bucket.instance","name":"tkscyf-38:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.38","timestamp":"2021-02-11T09:50:56.508532698Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_M7gYPbaihLb1WY3hhZTdAtg","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037056.510275_353.1","section":"bucket","name":"tkscyf-38","timestamp":"2021-02-11T09:50:56.510275102Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0RzXHcbVo20pBILfHDUWraY","ver":1},"status":{"status":"write"}}},{"id":"1_1613037056.514482_354.1","section":"bucket","name":"tkscyf-38","timestamp":"2021-02-11T09:50:56.514482141Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0RzXHcbVo20pBILfHDUWraY","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037057.908443_355.1","section":"bucket.instance","name":"tkscyf-39:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.39","timestamp":"2021-02-11T09:50:57.908443576Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_94HMkmvhzlZXLQvruAOVtAJ","ver":1},"status":{"status":"write"}}},{"id":"1_1613037057.914397_356.1","section":"bucket.instance","name":"tkscyf-39:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.39","timestamp":"2021-02-11T09:50:57.914397041Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_94HMkmvhzlZXLQvruAOVtAJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037057.916715_357.1","section":"bucket","name":"tkscyf-39","timestamp":"2021-02-11T09:50:57.916715595Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wsKJ1QOckGXWesrqOCyo1WG","ver":1},"status":{"status":"write"}}},{"id":"1_1613037057.923356_358.1","section":"bucket","name":"tkscyf-39","timestamp":"2021-02-11T09:50:57.923356621Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wsKJ1QOckGXWesrqOCyo1WG","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037057.945478_359.1","section":"bucket.instance","name":"tkscyf-40:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.40","timestamp":"2021-02-11T09:50:57.945478017Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pg3ahh-4pO-_3XyceN-gWsQ","ver":1},"status":{"status":"write"}}},{"id":"1_1613037057.949753_360.1","section":"bucket.instance","name":"tkscyf-40:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.40","timestamp":"2021-02-11T09:50:57.949753049Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pg3ahh-4pO-_3XyceN-gWsQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037057.951444_361.1","section":"bucket","name":"tkscyf-40","timestamp":"2021-02-11T09:50:57.951444977Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ej1h_jFuYjzyoRvb-1C7RAf","ver":1},"status":{"status":"write"}}},{"id":"1_1613037057.955584_362.1","section":"bucket","name":"tkscyf-40","timestamp":"2021-02-11T09:50:57.955584939Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ej1h_jFuYjzyoRvb-1C7RAf","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037057.996014_363.1","section":"bucket.instance","name":"tkscyf-41:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.41","timestamp":"2021-02-11T09:50:57.996014692Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_I9DAcVu7OInplRV1Ta3wruY","ver":1},"status":{"status":"write"}}},{"id":"1_1613037058.000113_364.1","section":"bucket.instance","name":"tkscyf-41:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.41","timestamp":"2021-02-11T09:50:58.000113498Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_I9DAcVu7OInplRV1Ta3wruY","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037058.001757_365.1","section":"bucket","name":"tkscyf-41","timestamp":"2021-02-11T09:50:58.001757095Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_R0j0lRoRH3ZszkT7rysin2m","ver":1},"status":{"status":"write"}}},{"id":"1_1613037058.005761_366.1","section":"bucket","name":"tkscyf-41","timestamp":"2021-02-11T09:50:58.005761416Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_R0j0lRoRH3ZszkT7rysin2m","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036843.308060_40.1","section":"bucket.instance","name":"tkscyf-1:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.1","timestamp":"2021-02-11T09:47:23.308060874Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vJcQgK-3XDRzzGYAO1_aHfA","ver":1},"status":{"status":"write"}}},{"id":"1_1613036843.334677_41.1","section":"bucket.instance","name":"tkscyf-1:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.1","timestamp":"2021-02-11T09:47:23.334677253Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vJcQgK-3XDRzzGYAO1_aHfA","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036843.341061_42.1","section":"bucket","name":"tkscyf-1","timestamp":"2021-02-11T09:47:23.341061810Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_so1GMpfrEZgmi_sTYZSAr8H","ver":1},"status":{"status":"write"}}},{"id":"1_1613036843.345437_43.1","section":"bucket","name":"tkscyf-1","timestamp":"2021-02-11T09:47:23.345437347Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_so1GMpfrEZgmi_sTYZSAr8H","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036843.366232_44.1","section":"bucket.instance","name":"tkscyf-2:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.2","timestamp":"2021-02-11T09:47:23.366232598Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MZ24WdPDpXrrqQPnafXRqiN","ver":1},"status":{"status":"write"}}},{"id":"1_1613036843.371311_45.1","section":"bucket.instance","name":"tkscyf-2:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.2","timestamp":"2021-02-11T09:47:23.371311720Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MZ24WdPDpXrrqQPnafXRqiN","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036843.373088_46.1","section":"bucket","name":"tkscyf-2","timestamp":"2021-02-11T09:47:23.373088017Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5VYuSj_-KrHdRO5ccn8Xe_y","ver":1},"status":{"status":"write"}}},{"id":"1_1613036843.378050_47.1","section":"bucket","name":"tkscyf-2","timestamp":"2021-02-11T09:47:23.378050996Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5VYuSj_-KrHdRO5ccn8Xe_y","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036847.401028_48.1","section":"bucket.instance","name":"tkscyf-3:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.3","timestamp":"2021-02-11T09:47:27.401028055Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ymULWMaifA40PYB81mtiqKO","ver":1},"status":{"status":"write"}}},{"id":"1_1613036847.406630_49.1","section":"bucket.instance","name":"tkscyf-3:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.3","timestamp":"2021-02-11T09:47:27.406630738Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ymULWMaifA40PYB81mtiqKO","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036847.408599_50.1","section":"bucket","name":"tkscyf-3","timestamp":"2021-02-11T09:47:27.408599295Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8F2ONXee867moT-prh9FMvz","ver":1},"status":{"status":"write"}}},{"id":"1_1613036847.414041_51.1","section":"bucket","name":"tkscyf-3","timestamp":"2021-02-11T09:47:27.414041891Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8F2ONXee867moT-prh9FMvz","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036848.515900_52.1","section":"bucket.instance","name":"tkscyf-4:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.4","timestamp":"2021-02-11T09:47:28.515900958Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OC0h4nNzRptiVogF5DDSlbl","ver":1},"status":{"status":"write"}}},{"id":"1_1613036848.521086_53.1","section":"bucket.instance","name":"tkscyf-4:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.4","timestamp":"2021-02-11T09:47:28.521086371Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OC0h4nNzRptiVogF5DDSlbl","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036848.523142_54.1","section":"bucket","name":"tkscyf-4","timestamp":"2021-02-11T09:47:28.523142343Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lzBpaQfMhBTasrckHQnxsrH","ver":1},"status":{"status":"write"}}},{"id":"1_1613036848.528085_55.1","section":"bucket","name":"tkscyf-4","timestamp":"2021-02-11T09:47:28.528085857Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lzBpaQfMhBTasrckHQnxsrH","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036848.545712_56.1","section":"bucket.instance","name":"tkscyf-5:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.5","timestamp":"2021-02-11T09:47:28.545712519Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pizM_4Vxb6ga5d7zQm5cup4","ver":1},"status":{"status":"write"}}},{"id":"1_1613036848.551316_57.1","section":"bucket.instance","name":"tkscyf-5:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.5","timestamp":"2021-02-11T09:47:28.551316982Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pizM_4Vxb6ga5d7zQm5cup4","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036848.553108_58.1","section":"bucket","name":"tkscyf-5","timestamp":"2021-02-11T09:47:28.553108370Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__btqQaGnBiVqesmRij4KpZu","ver":1},"status":{"status":"write"}}},{"id":"1_1613036848.557871_59.1","section":"bucket","name":"tkscyf-5","timestamp":"2021-02-11T09:47:28.557871264Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__btqQaGnBiVqesmRij4KpZu","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036848.598426_60.1","section":"bucket.instance","name":"tkscyf-6:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.6","timestamp":"2021-02-11T09:47:28.598426655Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MXFYvcf4zPQLqC1AyVkpK27","ver":1},"status":{"status":"write"}}},{"id":"1_1613036848.607442_61.1","section":"bucket.instance","name":"tkscyf-6:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.6","timestamp":"2021-02-11T09:47:28.607442762Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MXFYvcf4zPQLqC1AyVkpK27","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036848.609097_62.1","section":"bucket","name":"tkscyf-6","timestamp":"2021-02-11T09:47:28.609097002Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_27D4N2MBNgnn-CUkGMJjydq","ver":1},"status":{"status":"write"}}},{"id":"1_1613036848.613340_63.1","section":"bucket","name":"tkscyf-6","timestamp":"2021-02-11T09:47:28.613340684Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_27D4N2MBNgnn-CUkGMJjydq","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036851.051651_64.1","section":"bucket.instance","name":"tkscyf-7:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.7","timestamp":"2021-02-11T09:47:31.051651058Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qdftsvlxuD5g0Vn1ajzNmyo","ver":1},"status":{"status":"write"}}},{"id":"1_1613036851.056767_65.1","section":"bucket.instance","name":"tkscyf-7:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.7","timestamp":"2021-02-11T09:47:31.056767922Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qdftsvlxuD5g0Vn1ajzNmyo","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036851.058594_66.1","section":"bucket","name":"tkscyf-7","timestamp":"2021-02-11T09:47:31.058594967Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FCpy5wrdCKP58fp9iJu68WQ","ver":1},"status":{"status":"write"}}},{"id":"1_1613036851.063396_67.1","section":"bucket","name":"tkscyf-7","timestamp":"2021-02-11T09:47:31.063396372Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FCpy5wrdCKP58fp9iJu68WQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036851.082324_68.1","section":"bucket.instance","name":"tkscyf-8:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.8","timestamp":"2021-02-11T09:47:31.082324615Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZgPd4GGXu--UcP5ytH49TD4","ver":1},"status":{"status":"write"}}},{"id":"1_1613036851.087320_69.1","section":"bucket.instance","name":"tkscyf-8:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.8","timestamp":"2021-02-11T09:47:31.087320996Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZgPd4GGXu--UcP5ytH49TD4","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036851.089023_70.1","section":"bucket","name":"tkscyf-8","timestamp":"2021-02-11T09:47:31.089023916Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MfurQNSSeZct4L9MZ9BG8Gr","ver":1},"status":{"status":"write"}}},{"id":"1_1613036851.093557_71.1","section":"bucket","name":"tkscyf-8","timestamp":"2021-02-11T09:47:31.093557563Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MfurQNSSeZct4L9MZ9BG8Gr","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036851.133487_72.1","section":"bucket.instance","name":"tkscyf-9:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.9","timestamp":"2021-02-11T09:47:31.133487717Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gym2dhV8yR1PgdklxT3K2xe","ver":1},"status":{"status":"write"}}},{"id":"1_1613036851.138269_73.1","section":"bucket.instance","name":"tkscyf-9:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.9","timestamp":"2021-02-11T09:47:31.138269287Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gym2dhV8yR1PgdklxT3K2xe","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036851.139876_74.1","section":"bucket","name":"tkscyf-9","timestamp":"2021-02-11T09:47:31.139876187Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_p0bbqr2Y572m56xpGPFsiZ-","ver":1},"status":{"status":"write"}}},{"id":"1_1613036851.144668_75.1","section":"bucket","name":"tkscyf-9","timestamp":"2021-02-11T09:47:31.144668547Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_p0bbqr2Y572m56xpGPFsiZ-","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036863.874762_76.1","section":"bucket","name":"tkscyf-7","timestamp":"2021-02-11T09:47:43.874762601Z","data":{"read_version":{"tag":"_FCpy5wrdCKP58fp9iJu68WQ","ver":1},"write_version":{"tag":"_FCpy5wrdCKP58fp9iJu68WQ","ver":2},"status":{"status":"remove"}}},{"id":"1_1613036863.880950_77.1","section":"bucket","name":"tkscyf-7","timestamp":"2021-02-11T09:47:43.880950937Z","data":{"read_version":{"tag":"_FCpy5wrdCKP58fp9iJu68WQ","ver":1},"write_version":{"tag":"_FCpy5wrdCKP58fp9iJu68WQ","ver":2},"status":{"status":"complete"}}},{"id":"1_1613036863.956967_78.1","section":"bucket","name":"tkscyf-8","timestamp":"2021-02-11T09:47:43.956967900Z","data":{"read_version":{"tag":"_MfurQNSSeZct4L9MZ9BG8Gr","ver":1},"write_version":{"tag":"_MfurQNSSeZct4L9MZ9BG8Gr","ver":2},"status":{"status":"remove"}}},{"id":"1_1613036863.963661_79.1","section":"bucket","name":"tkscyf-8","timestamp":"2021-02-11T09:47:43.963661824Z","data":{"read_version":{"tag":"_MfurQNSSeZct4L9MZ9BG8Gr","ver":1},"write_version":{"tag":"_MfurQNSSeZct4L9MZ9BG8Gr","ver":2},"status":{"status":"complete"}}},{"id":"1_1613036864.074054_80.1","section":"bucket","name":"tkscyf-9","timestamp":"2021-02-11T09:47:44.074054305Z","data":{"read_version":{"tag":"_p0bbqr2Y572m56xpGPFsiZ-","ver":1},"write_version":{"tag":"_p0bbqr2Y572m56xpGPFsiZ-","ver":2},"status":{"status":"remove"}}},{"id":"1_1613036864.081584_81.1","section":"bucket","name":"tkscyf-9","timestamp":"2021-02-11T09:47:44.081584235Z","data":{"read_version":{"tag":"_p0bbqr2Y572m56xpGPFsiZ-","ver":1},"write_version":{"tag":"_p0bbqr2Y572m56xpGPFsiZ-","ver":2},"status":{"status":"complete"}}}] 2021-02-11T09:53:14.767 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:14.767 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:14.767 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:14.807 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:14.807 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:15.039 INFO:teuthology.orchestra.run.smithi094.stdout:[{"id":"1_1613036708.088554_48.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2021-02-11T09:45:08.088554566Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RA3OvQQVI2OcioTs6vG61ts","ver":1},"status":{"status":"write"}}},{"id":"1_1613036708.096437_52.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2021-02-11T09:45:08.096437057Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RA3OvQQVI2OcioTs6vG61ts","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036865.186535_60.1","section":"bucket.instance","name":"tkscyf-10:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.10","timestamp":"2021-02-11T09:47:45.186535085Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BOQUev-Msz6Gg8HQx3I-Uw_","ver":1},"status":{"status":"write"}}},{"id":"1_1613036865.191608_62.1","section":"bucket.instance","name":"tkscyf-10:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.10","timestamp":"2021-02-11T09:47:45.191608247Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BOQUev-Msz6Gg8HQx3I-Uw_","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036865.193557_63.1","section":"bucket","name":"tkscyf-10","timestamp":"2021-02-11T09:47:45.193557933Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fhFnaBfHBt4OCc3A0Cy3U0l","ver":1},"status":{"status":"write"}}},{"id":"1_1613036865.198089_64.1","section":"bucket","name":"tkscyf-10","timestamp":"2021-02-11T09:47:45.198089937Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fhFnaBfHBt4OCc3A0Cy3U0l","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036865.217116_66.1","section":"bucket.instance","name":"tkscyf-11:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.11","timestamp":"2021-02-11T09:47:45.217116956Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_x_UhJIyltXoCF5sGbvGrvWs","ver":1},"status":{"status":"write"}}},{"id":"1_1613036865.222221_67.1","section":"bucket.instance","name":"tkscyf-11:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.11","timestamp":"2021-02-11T09:47:45.222221108Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_x_UhJIyltXoCF5sGbvGrvWs","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036865.224285_68.1","section":"bucket","name":"tkscyf-11","timestamp":"2021-02-11T09:47:45.224285083Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FRR9uzZkCcSEP86_Kp419wb","ver":1},"status":{"status":"write"}}},{"id":"1_1613036865.228963_69.1","section":"bucket","name":"tkscyf-11","timestamp":"2021-02-11T09:47:45.228963299Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FRR9uzZkCcSEP86_Kp419wb","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036865.267357_70.1","section":"bucket.instance","name":"tkscyf-12:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.12","timestamp":"2021-02-11T09:47:45.267357171Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QtLjmo_4M7YVwSusOVuvKW7","ver":1},"status":{"status":"write"}}},{"id":"1_1613036865.272260_71.1","section":"bucket.instance","name":"tkscyf-12:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.12","timestamp":"2021-02-11T09:47:45.272260535Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QtLjmo_4M7YVwSusOVuvKW7","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036865.273997_72.1","section":"bucket","name":"tkscyf-12","timestamp":"2021-02-11T09:47:45.273997074Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9ZVFThqlMn2Lww6c1ZLx1-O","ver":1},"status":{"status":"write"}}},{"id":"1_1613036865.278468_73.1","section":"bucket","name":"tkscyf-12","timestamp":"2021-02-11T09:47:45.278468126Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9ZVFThqlMn2Lww6c1ZLx1-O","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036885.857139_178.1","section":"bucket.instance","name":"tkscyf-13:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.13","timestamp":"2021-02-11T09:48:05.857139297Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LjOnUQz6uMNf3yxGrmeonuy","ver":1},"status":{"status":"write"}}},{"id":"1_1613036885.861989_179.1","section":"bucket.instance","name":"tkscyf-13:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.13","timestamp":"2021-02-11T09:48:05.861989993Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LjOnUQz6uMNf3yxGrmeonuy","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036885.863884_180.1","section":"bucket","name":"tkscyf-13","timestamp":"2021-02-11T09:48:05.863884773Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f0vlqtQxcyVJgs3vWIVlMY5","ver":1},"status":{"status":"write"}}},{"id":"1_1613036885.868524_181.1","section":"bucket","name":"tkscyf-13","timestamp":"2021-02-11T09:48:05.868524702Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f0vlqtQxcyVJgs3vWIVlMY5","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036885.887239_182.1","section":"bucket.instance","name":"tkscyf-14:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.14","timestamp":"2021-02-11T09:48:05.887239288Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Cb1zieCDGGuK7mRxM9X7cOY","ver":1},"status":{"status":"write"}}},{"id":"1_1613036885.892744_183.1","section":"bucket.instance","name":"tkscyf-14:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.14","timestamp":"2021-02-11T09:48:05.892744502Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Cb1zieCDGGuK7mRxM9X7cOY","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036885.894666_184.1","section":"bucket","name":"tkscyf-14","timestamp":"2021-02-11T09:48:05.894666449Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0d8lwgEigMOkq-FSumkqW71","ver":1},"status":{"status":"write"}}},{"id":"1_1613036885.900682_185.1","section":"bucket","name":"tkscyf-14","timestamp":"2021-02-11T09:48:05.900682555Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0d8lwgEigMOkq-FSumkqW71","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036885.942706_186.1","section":"bucket.instance","name":"tkscyf-15:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.15","timestamp":"2021-02-11T09:48:05.942706611Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-hfQZ5lWiy4o6NP7OEZNQ0h","ver":1},"status":{"status":"write"}}},{"id":"1_1613036885.947464_187.1","section":"bucket.instance","name":"tkscyf-15:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.15","timestamp":"2021-02-11T09:48:05.947464351Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-hfQZ5lWiy4o6NP7OEZNQ0h","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036885.949248_188.1","section":"bucket","name":"tkscyf-15","timestamp":"2021-02-11T09:48:05.949248580Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4gdcmHzvr3iqrlpLrZQpyg1","ver":1},"status":{"status":"write"}}},{"id":"1_1613036885.953989_189.1","section":"bucket","name":"tkscyf-15","timestamp":"2021-02-11T09:48:05.953989244Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4gdcmHzvr3iqrlpLrZQpyg1","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036899.854288_194.1","section":"bucket.instance","name":"tkscyf-16:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.16","timestamp":"2021-02-11T09:48:19.854288478Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BjHGVIhMvUslBcZQfe5MTso","ver":1},"status":{"status":"write"}}},{"id":"1_1613036899.859377_195.1","section":"bucket.instance","name":"tkscyf-16:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.16","timestamp":"2021-02-11T09:48:19.859377386Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BjHGVIhMvUslBcZQfe5MTso","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036899.861235_196.1","section":"bucket","name":"tkscyf-16","timestamp":"2021-02-11T09:48:19.861235489Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_mBUX065ZmzTTf0GD1alWPkY","ver":1},"status":{"status":"write"}}},{"id":"1_1613036899.866335_197.1","section":"bucket","name":"tkscyf-16","timestamp":"2021-02-11T09:48:19.866335427Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_mBUX065ZmzTTf0GD1alWPkY","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036899.888709_198.1","section":"bucket.instance","name":"tkscyf-17:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.17","timestamp":"2021-02-11T09:48:19.888709670Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_upQr8yx99-au6_AWkKxYuHu","ver":1},"status":{"status":"write"}}},{"id":"1_1613036899.894272_199.1","section":"bucket.instance","name":"tkscyf-17:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.17","timestamp":"2021-02-11T09:48:19.894272943Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_upQr8yx99-au6_AWkKxYuHu","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036899.896430_200.1","section":"bucket","name":"tkscyf-17","timestamp":"2021-02-11T09:48:19.896430934Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_EVefVFc5TujdFr9n6vuddup","ver":1},"status":{"status":"write"}}},{"id":"1_1613036899.901399_201.1","section":"bucket","name":"tkscyf-17","timestamp":"2021-02-11T09:48:19.901399970Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_EVefVFc5TujdFr9n6vuddup","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036899.942394_202.1","section":"bucket.instance","name":"tkscyf-18:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.18","timestamp":"2021-02-11T09:48:19.942394398Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eLelEHOSgz_5RWVJgejlcwE","ver":1},"status":{"status":"write"}}},{"id":"1_1613036899.947586_203.1","section":"bucket.instance","name":"tkscyf-18:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.18","timestamp":"2021-02-11T09:48:19.947586263Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eLelEHOSgz_5RWVJgejlcwE","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036899.949797_204.1","section":"bucket","name":"tkscyf-18","timestamp":"2021-02-11T09:48:19.949797527Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nnXNYQhXpslLZMkcMsbaLYW","ver":1},"status":{"status":"write"}}},{"id":"1_1613036899.954577_205.1","section":"bucket","name":"tkscyf-18","timestamp":"2021-02-11T09:48:19.954577937Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nnXNYQhXpslLZMkcMsbaLYW","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036899.976313_206.1","section":"bucket.instance","name":"tkscyf-16:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.16","timestamp":"2021-02-11T09:48:19.976313350Z","data":{"read_version":{"tag":"_BjHGVIhMvUslBcZQfe5MTso","ver":1},"write_version":{"tag":"_BjHGVIhMvUslBcZQfe5MTso","ver":2},"status":{"status":"write"}}},{"id":"1_1613036899.981307_207.1","section":"bucket.instance","name":"tkscyf-16:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.16","timestamp":"2021-02-11T09:48:19.981307840Z","data":{"read_version":{"tag":"_BjHGVIhMvUslBcZQfe5MTso","ver":1},"write_version":{"tag":"_BjHGVIhMvUslBcZQfe5MTso","ver":2},"status":{"status":"complete"}}},{"id":"1_1613036899.986228_208.1","section":"bucket.instance","name":"tkscyf-17:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.17","timestamp":"2021-02-11T09:48:19.986228789Z","data":{"read_version":{"tag":"_upQr8yx99-au6_AWkKxYuHu","ver":1},"write_version":{"tag":"_upQr8yx99-au6_AWkKxYuHu","ver":2},"status":{"status":"write"}}},{"id":"1_1613036899.991766_209.1","section":"bucket.instance","name":"tkscyf-17:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.17","timestamp":"2021-02-11T09:48:19.991766410Z","data":{"read_version":{"tag":"_upQr8yx99-au6_AWkKxYuHu","ver":1},"write_version":{"tag":"_upQr8yx99-au6_AWkKxYuHu","ver":2},"status":{"status":"complete"}}},{"id":"1_1613036900.005531_210.1","section":"bucket.instance","name":"tkscyf-18:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.18","timestamp":"2021-02-11T09:48:20.005531886Z","data":{"read_version":{"tag":"_eLelEHOSgz_5RWVJgejlcwE","ver":1},"write_version":{"tag":"_eLelEHOSgz_5RWVJgejlcwE","ver":2},"status":{"status":"write"}}},{"id":"1_1613036900.015316_211.1","section":"bucket.instance","name":"tkscyf-18:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.18","timestamp":"2021-02-11T09:48:20.015316565Z","data":{"read_version":{"tag":"_eLelEHOSgz_5RWVJgejlcwE","ver":1},"write_version":{"tag":"_eLelEHOSgz_5RWVJgejlcwE","ver":2},"status":{"status":"complete"}}},{"id":"1_1613036950.507710_227.1","section":"bucket.instance","name":"tkscyf-19:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.19","timestamp":"2021-02-11T09:49:10.507710607Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_G8JIgZ6gi-tWTxNuGHdxLfV","ver":1},"status":{"status":"write"}}},{"id":"1_1613036950.512728_228.1","section":"bucket.instance","name":"tkscyf-19:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.19","timestamp":"2021-02-11T09:49:10.512728298Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_G8JIgZ6gi-tWTxNuGHdxLfV","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036950.514564_229.1","section":"bucket","name":"tkscyf-19","timestamp":"2021-02-11T09:49:10.514564712Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rUH9m8tDhvujVPRfyQMubOa","ver":1},"status":{"status":"write"}}},{"id":"1_1613036950.519470_230.1","section":"bucket","name":"tkscyf-19","timestamp":"2021-02-11T09:49:10.519470166Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rUH9m8tDhvujVPRfyQMubOa","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036950.525237_231.1","section":"bucket.instance","name":"tkscyf-19:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.19","timestamp":"2021-02-11T09:49:10.525237169Z","data":{"read_version":{"tag":"_G8JIgZ6gi-tWTxNuGHdxLfV","ver":1},"write_version":{"tag":"_G8JIgZ6gi-tWTxNuGHdxLfV","ver":2},"status":{"status":"write"}}},{"id":"1_1613036950.530761_232.1","section":"bucket.instance","name":"tkscyf-19:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.19","timestamp":"2021-02-11T09:49:10.530761661Z","data":{"read_version":{"tag":"_G8JIgZ6gi-tWTxNuGHdxLfV","ver":1},"write_version":{"tag":"_G8JIgZ6gi-tWTxNuGHdxLfV","ver":2},"status":{"status":"complete"}}},{"id":"1_1613036998.267812_238.1","section":"bucket.instance","name":"tkscyf-20:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.20","timestamp":"2021-02-11T09:49:58.267812499Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3X25hzqCxgrngeWBjXAq3MW","ver":1},"status":{"status":"write"}}},{"id":"1_1613036998.273260_239.1","section":"bucket.instance","name":"tkscyf-20:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.20","timestamp":"2021-02-11T09:49:58.273260665Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3X25hzqCxgrngeWBjXAq3MW","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036998.275321_240.1","section":"bucket","name":"tkscyf-20","timestamp":"2021-02-11T09:49:58.275321795Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GSd-2J0lHNWhUfK9gtbzsaz","ver":1},"status":{"status":"write"}}},{"id":"1_1613036998.280111_241.1","section":"bucket","name":"tkscyf-20","timestamp":"2021-02-11T09:49:58.280111713Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GSd-2J0lHNWhUfK9gtbzsaz","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037007.140764_243.1","section":"bucket.instance","name":"tkscyf-20:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.20","timestamp":"2021-02-11T09:50:07.140764026Z","data":{"read_version":{"tag":"_3X25hzqCxgrngeWBjXAq3MW","ver":1},"write_version":{"tag":"_3X25hzqCxgrngeWBjXAq3MW","ver":2},"status":{"status":"write"}}},{"id":"1_1613037007.150589_244.1","section":"bucket.instance","name":"tkscyf-20:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.20","timestamp":"2021-02-11T09:50:07.150589879Z","data":{"read_version":{"tag":"_3X25hzqCxgrngeWBjXAq3MW","ver":1},"write_version":{"tag":"_3X25hzqCxgrngeWBjXAq3MW","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037012.304895_245.1","section":"bucket.instance","name":"tkscyf-20:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.20","timestamp":"2021-02-11T09:50:12.304895189Z","data":{"read_version":{"tag":"_3X25hzqCxgrngeWBjXAq3MW","ver":2},"write_version":{"tag":"_3X25hzqCxgrngeWBjXAq3MW","ver":3},"status":{"status":"write"}}},{"id":"1_1613037012.316265_246.1","section":"bucket.instance","name":"tkscyf-20:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.20","timestamp":"2021-02-11T09:50:12.316265099Z","data":{"read_version":{"tag":"_3X25hzqCxgrngeWBjXAq3MW","ver":2},"write_version":{"tag":"_3X25hzqCxgrngeWBjXAq3MW","ver":3},"status":{"status":"complete"}}},{"id":"1_1613037016.767451_247.1","section":"bucket.instance","name":"tkscyf-21:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.21","timestamp":"2021-02-11T09:50:16.767451636Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_usNhex5LVJs2K5fhCTS-E9B","ver":1},"status":{"status":"write"}}},{"id":"1_1613037016.772791_248.1","section":"bucket.instance","name":"tkscyf-21:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.21","timestamp":"2021-02-11T09:50:16.772791796Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_usNhex5LVJs2K5fhCTS-E9B","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037016.774734_249.1","section":"bucket","name":"tkscyf-21","timestamp":"2021-02-11T09:50:16.774734911Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_oS8fsXNPnQ3YSGsw6yEPI6c","ver":1},"status":{"status":"write"}}},{"id":"1_1613037016.779555_250.1","section":"bucket","name":"tkscyf-21","timestamp":"2021-02-11T09:50:16.779555063Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_oS8fsXNPnQ3YSGsw6yEPI6c","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037016.800629_251.1","section":"bucket.instance","name":"tkscyf-22:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.22","timestamp":"2021-02-11T09:50:16.800629460Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Q5XnU57eMsSQj5AxurjrM1s","ver":1},"status":{"status":"write"}}},{"id":"1_1613037016.807098_252.1","section":"bucket.instance","name":"tkscyf-22:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.22","timestamp":"2021-02-11T09:50:16.807098120Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Q5XnU57eMsSQj5AxurjrM1s","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037016.809071_253.1","section":"bucket","name":"tkscyf-22","timestamp":"2021-02-11T09:50:16.809071480Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Jcw-M7gbUwOe2t495PGK9vo","ver":1},"status":{"status":"write"}}},{"id":"1_1613037016.813672_254.1","section":"bucket","name":"tkscyf-22","timestamp":"2021-02-11T09:50:16.813672741Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Jcw-M7gbUwOe2t495PGK9vo","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037016.860072_255.1","section":"bucket.instance","name":"tkscyf-23:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.23","timestamp":"2021-02-11T09:50:16.860072689Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GCLr8VFsBOKFg5ZuZUx_GvZ","ver":1},"status":{"status":"write"}}},{"id":"1_1613037016.866566_256.1","section":"bucket.instance","name":"tkscyf-23:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.23","timestamp":"2021-02-11T09:50:16.866566184Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GCLr8VFsBOKFg5ZuZUx_GvZ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037016.869661_257.1","section":"bucket","name":"tkscyf-23","timestamp":"2021-02-11T09:50:16.869661986Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8HFWSnbWVi7UHPWj_xF4vXf","ver":1},"status":{"status":"write"}}},{"id":"1_1613037016.879001_258.1","section":"bucket","name":"tkscyf-23","timestamp":"2021-02-11T09:50:16.879001731Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8HFWSnbWVi7UHPWj_xF4vXf","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037016.897902_259.1","section":"bucket.instance","name":"tkscyf-21:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.21","timestamp":"2021-02-11T09:50:16.897902097Z","data":{"read_version":{"tag":"_usNhex5LVJs2K5fhCTS-E9B","ver":1},"write_version":{"tag":"_usNhex5LVJs2K5fhCTS-E9B","ver":2},"status":{"status":"write"}}},{"id":"1_1613037016.902295_260.1","section":"bucket.instance","name":"tkscyf-21:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.21","timestamp":"2021-02-11T09:50:16.902295553Z","data":{"read_version":{"tag":"_usNhex5LVJs2K5fhCTS-E9B","ver":1},"write_version":{"tag":"_usNhex5LVJs2K5fhCTS-E9B","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037016.906251_261.1","section":"bucket.instance","name":"tkscyf-22:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.22","timestamp":"2021-02-11T09:50:16.906251476Z","data":{"read_version":{"tag":"_Q5XnU57eMsSQj5AxurjrM1s","ver":1},"write_version":{"tag":"_Q5XnU57eMsSQj5AxurjrM1s","ver":2},"status":{"status":"write"}}},{"id":"1_1613037016.911598_262.1","section":"bucket.instance","name":"tkscyf-22:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.22","timestamp":"2021-02-11T09:50:16.911598777Z","data":{"read_version":{"tag":"_Q5XnU57eMsSQj5AxurjrM1s","ver":1},"write_version":{"tag":"_Q5XnU57eMsSQj5AxurjrM1s","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037016.926597_263.1","section":"bucket.instance","name":"tkscyf-23:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.23","timestamp":"2021-02-11T09:50:16.926597890Z","data":{"read_version":{"tag":"_GCLr8VFsBOKFg5ZuZUx_GvZ","ver":1},"write_version":{"tag":"_GCLr8VFsBOKFg5ZuZUx_GvZ","ver":2},"status":{"status":"write"}}},{"id":"1_1613037016.931312_264.1","section":"bucket.instance","name":"tkscyf-23:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.23","timestamp":"2021-02-11T09:50:16.931312694Z","data":{"read_version":{"tag":"_GCLr8VFsBOKFg5ZuZUx_GvZ","ver":1},"write_version":{"tag":"_GCLr8VFsBOKFg5ZuZUx_GvZ","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037035.238424_265.1","section":"bucket.instance","name":"tkscyf-24:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.24","timestamp":"2021-02-11T09:50:35.238424865Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bb0kUiSwNRO-U3TZ4E1z9m5","ver":1},"status":{"status":"write"}}},{"id":"1_1613037035.243640_266.1","section":"bucket.instance","name":"tkscyf-24:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.24","timestamp":"2021-02-11T09:50:35.243640241Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bb0kUiSwNRO-U3TZ4E1z9m5","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037035.245475_267.1","section":"bucket","name":"tkscyf-24","timestamp":"2021-02-11T09:50:35.245475848Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7Yg_eLnx93F7TXgKcKQAxT5","ver":1},"status":{"status":"write"}}},{"id":"1_1613037035.250290_268.1","section":"bucket","name":"tkscyf-24","timestamp":"2021-02-11T09:50:35.250290672Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7Yg_eLnx93F7TXgKcKQAxT5","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037035.274261_269.1","section":"bucket.instance","name":"tkscyf-25:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.25","timestamp":"2021-02-11T09:50:35.274261140Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1nnotJE0A2l-djRVSBIWs9E","ver":1},"status":{"status":"write"}}},{"id":"1_1613037035.281072_270.1","section":"bucket.instance","name":"tkscyf-25:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.25","timestamp":"2021-02-11T09:50:35.281072011Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1nnotJE0A2l-djRVSBIWs9E","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037035.283085_271.1","section":"bucket","name":"tkscyf-25","timestamp":"2021-02-11T09:50:35.283085981Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PBWPxzfNM3rpuuSRbw9WWJ4","ver":1},"status":{"status":"write"}}},{"id":"1_1613037035.287331_272.1","section":"bucket","name":"tkscyf-25","timestamp":"2021-02-11T09:50:35.287331936Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PBWPxzfNM3rpuuSRbw9WWJ4","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037035.327494_273.1","section":"bucket.instance","name":"tkscyf-26:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.26","timestamp":"2021-02-11T09:50:35.327494695Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yBW2OnCxamt6jwtfytnSIYp","ver":1},"status":{"status":"write"}}},{"id":"1_1613037035.336740_274.1","section":"bucket.instance","name":"tkscyf-26:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.26","timestamp":"2021-02-11T09:50:35.336740442Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yBW2OnCxamt6jwtfytnSIYp","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037035.338820_275.1","section":"bucket","name":"tkscyf-26","timestamp":"2021-02-11T09:50:35.338820669Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GBS2JL6Meg0eeenzd3LFcRc","ver":1},"status":{"status":"write"}}},{"id":"1_1613037035.343318_276.1","section":"bucket","name":"tkscyf-26","timestamp":"2021-02-11T09:50:35.343318410Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GBS2JL6Meg0eeenzd3LFcRc","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037035.361989_277.1","section":"bucket.instance","name":"tkscyf-24:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.24","timestamp":"2021-02-11T09:50:35.361989484Z","data":{"read_version":{"tag":"_bb0kUiSwNRO-U3TZ4E1z9m5","ver":1},"write_version":{"tag":"_bb0kUiSwNRO-U3TZ4E1z9m5","ver":2},"status":{"status":"write"}}},{"id":"1_1613037035.367759_278.1","section":"bucket.instance","name":"tkscyf-24:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.24","timestamp":"2021-02-11T09:50:35.367759053Z","data":{"read_version":{"tag":"_bb0kUiSwNRO-U3TZ4E1z9m5","ver":1},"write_version":{"tag":"_bb0kUiSwNRO-U3TZ4E1z9m5","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037035.371457_279.1","section":"bucket.instance","name":"tkscyf-24:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.24","timestamp":"2021-02-11T09:50:35.371457040Z","data":{"read_version":{"tag":"_bb0kUiSwNRO-U3TZ4E1z9m5","ver":2},"write_version":{"tag":"_bb0kUiSwNRO-U3TZ4E1z9m5","ver":3},"status":{"status":"write"}}},{"id":"1_1613037035.376968_280.1","section":"bucket.instance","name":"tkscyf-24:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.24","timestamp":"2021-02-11T09:50:35.376968483Z","data":{"read_version":{"tag":"_bb0kUiSwNRO-U3TZ4E1z9m5","ver":2},"write_version":{"tag":"_bb0kUiSwNRO-U3TZ4E1z9m5","ver":3},"status":{"status":"complete"}}},{"id":"1_1613037035.382067_281.1","section":"bucket.instance","name":"tkscyf-25:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.25","timestamp":"2021-02-11T09:50:35.382067509Z","data":{"read_version":{"tag":"_1nnotJE0A2l-djRVSBIWs9E","ver":1},"write_version":{"tag":"_1nnotJE0A2l-djRVSBIWs9E","ver":2},"status":{"status":"write"}}},{"id":"1_1613037035.388230_282.1","section":"bucket.instance","name":"tkscyf-25:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.25","timestamp":"2021-02-11T09:50:35.388230116Z","data":{"read_version":{"tag":"_1nnotJE0A2l-djRVSBIWs9E","ver":1},"write_version":{"tag":"_1nnotJE0A2l-djRVSBIWs9E","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037035.399024_283.1","section":"bucket.instance","name":"tkscyf-25:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.25","timestamp":"2021-02-11T09:50:35.399024419Z","data":{"read_version":{"tag":"_1nnotJE0A2l-djRVSBIWs9E","ver":2},"write_version":{"tag":"_1nnotJE0A2l-djRVSBIWs9E","ver":3},"status":{"status":"write"}}},{"id":"1_1613037035.406913_284.1","section":"bucket.instance","name":"tkscyf-25:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.25","timestamp":"2021-02-11T09:50:35.406913878Z","data":{"read_version":{"tag":"_1nnotJE0A2l-djRVSBIWs9E","ver":2},"write_version":{"tag":"_1nnotJE0A2l-djRVSBIWs9E","ver":3},"status":{"status":"complete"}}},{"id":"1_1613037035.418911_285.1","section":"bucket.instance","name":"tkscyf-26:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.26","timestamp":"2021-02-11T09:50:35.418911493Z","data":{"read_version":{"tag":"_yBW2OnCxamt6jwtfytnSIYp","ver":1},"write_version":{"tag":"_yBW2OnCxamt6jwtfytnSIYp","ver":2},"status":{"status":"write"}}},{"id":"1_1613037035.427825_286.1","section":"bucket.instance","name":"tkscyf-26:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.26","timestamp":"2021-02-11T09:50:35.427825538Z","data":{"read_version":{"tag":"_yBW2OnCxamt6jwtfytnSIYp","ver":1},"write_version":{"tag":"_yBW2OnCxamt6jwtfytnSIYp","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037035.439773_287.1","section":"bucket.instance","name":"tkscyf-26:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.26","timestamp":"2021-02-11T09:50:35.439773122Z","data":{"read_version":{"tag":"_yBW2OnCxamt6jwtfytnSIYp","ver":2},"write_version":{"tag":"_yBW2OnCxamt6jwtfytnSIYp","ver":3},"status":{"status":"write"}}},{"id":"1_1613037035.445782_288.1","section":"bucket.instance","name":"tkscyf-26:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.26","timestamp":"2021-02-11T09:50:35.445782025Z","data":{"read_version":{"tag":"_yBW2OnCxamt6jwtfytnSIYp","ver":2},"write_version":{"tag":"_yBW2OnCxamt6jwtfytnSIYp","ver":3},"status":{"status":"complete"}}},{"id":"1_1613037049.014616_289.1","section":"bucket.instance","name":"tkscyf-27:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.27","timestamp":"2021-02-11T09:50:49.014616484Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HPEAummOtXzD3iTYGsjL78L","ver":1},"status":{"status":"write"}}},{"id":"1_1613037049.020486_290.1","section":"bucket.instance","name":"tkscyf-27:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.27","timestamp":"2021-02-11T09:50:49.020486409Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HPEAummOtXzD3iTYGsjL78L","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037049.022446_291.1","section":"bucket","name":"tkscyf-27","timestamp":"2021-02-11T09:50:49.022446056Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HzrXfJNvB4DsZqVjRjsmkDL","ver":1},"status":{"status":"write"}}},{"id":"1_1613037049.026934_292.1","section":"bucket","name":"tkscyf-27","timestamp":"2021-02-11T09:50:49.026934689Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HzrXfJNvB4DsZqVjRjsmkDL","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037049.052544_293.1","section":"bucket.instance","name":"tkscyf-28:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.28","timestamp":"2021-02-11T09:50:49.052544751Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-VSdKrlRKUfuwloTfMI6Xuv","ver":1},"status":{"status":"write"}}},{"id":"1_1613037049.057739_294.1","section":"bucket.instance","name":"tkscyf-28:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.28","timestamp":"2021-02-11T09:50:49.057739941Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-VSdKrlRKUfuwloTfMI6Xuv","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037049.059973_295.1","section":"bucket","name":"tkscyf-28","timestamp":"2021-02-11T09:50:49.059973620Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-BX9bB1FIeOQc9AK3EsxPMy","ver":1},"status":{"status":"write"}}},{"id":"1_1613037049.064978_296.1","section":"bucket","name":"tkscyf-28","timestamp":"2021-02-11T09:50:49.064978209Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-BX9bB1FIeOQc9AK3EsxPMy","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037049.129163_297.1","section":"bucket.instance","name":"tkscyf-29:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.29","timestamp":"2021-02-11T09:50:49.129163449Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hyJCgZ3ZqhjZ2B82NLQtbFj","ver":1},"status":{"status":"write"}}},{"id":"1_1613037049.139199_298.1","section":"bucket.instance","name":"tkscyf-29:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.29","timestamp":"2021-02-11T09:50:49.139199855Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hyJCgZ3ZqhjZ2B82NLQtbFj","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037049.141005_299.1","section":"bucket","name":"tkscyf-29","timestamp":"2021-02-11T09:50:49.141005489Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1eRc33iNyGdkXUg6pUKi_qn","ver":1},"status":{"status":"write"}}},{"id":"1_1613037049.145434_300.1","section":"bucket","name":"tkscyf-29","timestamp":"2021-02-11T09:50:49.145434314Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1eRc33iNyGdkXUg6pUKi_qn","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037049.166096_301.1","section":"bucket.instance","name":"tkscyf-27:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.27","timestamp":"2021-02-11T09:50:49.166096762Z","data":{"read_version":{"tag":"_HPEAummOtXzD3iTYGsjL78L","ver":1},"write_version":{"tag":"_HPEAummOtXzD3iTYGsjL78L","ver":2},"status":{"status":"write"}}},{"id":"1_1613037049.171244_302.1","section":"bucket.instance","name":"tkscyf-27:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.27","timestamp":"2021-02-11T09:50:49.171244329Z","data":{"read_version":{"tag":"_HPEAummOtXzD3iTYGsjL78L","ver":1},"write_version":{"tag":"_HPEAummOtXzD3iTYGsjL78L","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037049.219869_303.1","section":"bucket.instance","name":"tkscyf-28:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.28","timestamp":"2021-02-11T09:50:49.219869180Z","data":{"read_version":{"tag":"_-VSdKrlRKUfuwloTfMI6Xuv","ver":1},"write_version":{"tag":"_-VSdKrlRKUfuwloTfMI6Xuv","ver":2},"status":{"status":"write"}}},{"id":"1_1613037049.230173_304.1","section":"bucket.instance","name":"tkscyf-28:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.28","timestamp":"2021-02-11T09:50:49.230173871Z","data":{"read_version":{"tag":"_-VSdKrlRKUfuwloTfMI6Xuv","ver":1},"write_version":{"tag":"_-VSdKrlRKUfuwloTfMI6Xuv","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037049.287108_305.1","section":"bucket.instance","name":"tkscyf-29:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.29","timestamp":"2021-02-11T09:50:49.287108951Z","data":{"read_version":{"tag":"_hyJCgZ3ZqhjZ2B82NLQtbFj","ver":1},"write_version":{"tag":"_hyJCgZ3ZqhjZ2B82NLQtbFj","ver":2},"status":{"status":"write"}}},{"id":"1_1613037049.300847_306.1","section":"bucket.instance","name":"tkscyf-29:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.29","timestamp":"2021-02-11T09:50:49.300847185Z","data":{"read_version":{"tag":"_hyJCgZ3ZqhjZ2B82NLQtbFj","ver":1},"write_version":{"tag":"_hyJCgZ3ZqhjZ2B82NLQtbFj","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037049.364299_307.1","section":"bucket.instance","name":"tkscyf-30:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.30","timestamp":"2021-02-11T09:50:49.364299749Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Es0JugEkT-f3o-riZAdqnrS","ver":1},"status":{"status":"write"}}},{"id":"1_1613037049.370623_308.1","section":"bucket.instance","name":"tkscyf-30:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.30","timestamp":"2021-02-11T09:50:49.370623916Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Es0JugEkT-f3o-riZAdqnrS","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037049.372900_309.1","section":"bucket","name":"tkscyf-30","timestamp":"2021-02-11T09:50:49.372900977Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0ipPDFF-oXK1PAXV9ImeRz8","ver":1},"status":{"status":"write"}}},{"id":"1_1613037049.377959_310.1","section":"bucket","name":"tkscyf-30","timestamp":"2021-02-11T09:50:49.377959725Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0ipPDFF-oXK1PAXV9ImeRz8","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037049.400272_311.1","section":"bucket.instance","name":"tkscyf-31:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.31","timestamp":"2021-02-11T09:50:49.400272843Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_J_JeLJIs-yKAxn3I-jH6tun","ver":1},"status":{"status":"write"}}},{"id":"1_1613037049.406204_312.1","section":"bucket.instance","name":"tkscyf-31:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.31","timestamp":"2021-02-11T09:50:49.406204831Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_J_JeLJIs-yKAxn3I-jH6tun","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037049.407984_313.1","section":"bucket","name":"tkscyf-31","timestamp":"2021-02-11T09:50:49.407984579Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AtSnm2w_MDdPl4ZTuW8BBX1","ver":1},"status":{"status":"write"}}},{"id":"1_1613037049.412626_314.1","section":"bucket","name":"tkscyf-31","timestamp":"2021-02-11T09:50:49.412626629Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AtSnm2w_MDdPl4ZTuW8BBX1","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037049.456098_315.1","section":"bucket.instance","name":"tkscyf-32:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.32","timestamp":"2021-02-11T09:50:49.456098996Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jg7YdCRiL4_wX7QgQ9ZEBCK","ver":1},"status":{"status":"write"}}},{"id":"1_1613037049.461384_316.1","section":"bucket.instance","name":"tkscyf-32:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.32","timestamp":"2021-02-11T09:50:49.461384431Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jg7YdCRiL4_wX7QgQ9ZEBCK","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037049.463339_317.1","section":"bucket","name":"tkscyf-32","timestamp":"2021-02-11T09:50:49.463339008Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eg1axhWTHrzA2NWrcdnReCD","ver":1},"status":{"status":"write"}}},{"id":"1_1613037049.468298_318.1","section":"bucket","name":"tkscyf-32","timestamp":"2021-02-11T09:50:49.468298585Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eg1axhWTHrzA2NWrcdnReCD","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037049.493719_319.1","section":"bucket.instance","name":"tkscyf-30:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.30","timestamp":"2021-02-11T09:50:49.493719070Z","data":{"read_version":{"tag":"_Es0JugEkT-f3o-riZAdqnrS","ver":1},"write_version":{"tag":"_Es0JugEkT-f3o-riZAdqnrS","ver":2},"status":{"status":"write"}}},{"id":"1_1613037049.498976_320.1","section":"bucket.instance","name":"tkscyf-30:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.30","timestamp":"2021-02-11T09:50:49.498976233Z","data":{"read_version":{"tag":"_Es0JugEkT-f3o-riZAdqnrS","ver":1},"write_version":{"tag":"_Es0JugEkT-f3o-riZAdqnrS","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037049.547179_321.1","section":"bucket.instance","name":"tkscyf-31:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.31","timestamp":"2021-02-11T09:50:49.547179938Z","data":{"read_version":{"tag":"_J_JeLJIs-yKAxn3I-jH6tun","ver":1},"write_version":{"tag":"_J_JeLJIs-yKAxn3I-jH6tun","ver":2},"status":{"status":"write"}}},{"id":"1_1613037049.552155_322.1","section":"bucket.instance","name":"tkscyf-31:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.31","timestamp":"2021-02-11T09:50:49.552155032Z","data":{"read_version":{"tag":"_J_JeLJIs-yKAxn3I-jH6tun","ver":1},"write_version":{"tag":"_J_JeLJIs-yKAxn3I-jH6tun","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037049.603090_323.1","section":"bucket.instance","name":"tkscyf-32:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.32","timestamp":"2021-02-11T09:50:49.603090607Z","data":{"read_version":{"tag":"_jg7YdCRiL4_wX7QgQ9ZEBCK","ver":1},"write_version":{"tag":"_jg7YdCRiL4_wX7QgQ9ZEBCK","ver":2},"status":{"status":"write"}}},{"id":"1_1613037049.608976_324.1","section":"bucket.instance","name":"tkscyf-32:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.32","timestamp":"2021-02-11T09:50:49.608976182Z","data":{"read_version":{"tag":"_jg7YdCRiL4_wX7QgQ9ZEBCK","ver":1},"write_version":{"tag":"_jg7YdCRiL4_wX7QgQ9ZEBCK","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037049.680125_325.1","section":"bucket.instance","name":"tkscyf-33:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.33","timestamp":"2021-02-11T09:50:49.680125618Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eZRAP_8ajxLAWs4hu7nqOwz","ver":1},"status":{"status":"write"}}},{"id":"1_1613037049.687075_326.1","section":"bucket.instance","name":"tkscyf-33:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.33","timestamp":"2021-02-11T09:50:49.687075481Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eZRAP_8ajxLAWs4hu7nqOwz","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037049.692386_327.1","section":"bucket","name":"tkscyf-33","timestamp":"2021-02-11T09:50:49.692386405Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-6dR6mEXl2aMVHaPbOLG9sm","ver":1},"status":{"status":"write"}}},{"id":"1_1613037049.699844_328.1","section":"bucket","name":"tkscyf-33","timestamp":"2021-02-11T09:50:49.699844848Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-6dR6mEXl2aMVHaPbOLG9sm","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037049.725396_329.1","section":"bucket.instance","name":"tkscyf-34:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.34","timestamp":"2021-02-11T09:50:49.725396604Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_k7kM2SM39DqWgu3NfoI4bqF","ver":1},"status":{"status":"write"}}},{"id":"1_1613037049.731707_330.1","section":"bucket.instance","name":"tkscyf-34:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.34","timestamp":"2021-02-11T09:50:49.731707348Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_k7kM2SM39DqWgu3NfoI4bqF","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037049.735696_331.1","section":"bucket","name":"tkscyf-34","timestamp":"2021-02-11T09:50:49.735696647Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_L016eB3Fp2Gc1WBb3ihrD9w","ver":1},"status":{"status":"write"}}},{"id":"1_1613037049.740587_332.1","section":"bucket","name":"tkscyf-34","timestamp":"2021-02-11T09:50:49.740587087Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_L016eB3Fp2Gc1WBb3ihrD9w","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037049.780026_333.1","section":"bucket.instance","name":"tkscyf-35:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.35","timestamp":"2021-02-11T09:50:49.780026803Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KosuwqaTc-yYnhDuNK50-jc","ver":1},"status":{"status":"write"}}},{"id":"1_1613037049.785523_334.1","section":"bucket.instance","name":"tkscyf-35:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.35","timestamp":"2021-02-11T09:50:49.785523728Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KosuwqaTc-yYnhDuNK50-jc","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037049.787540_335.1","section":"bucket","name":"tkscyf-35","timestamp":"2021-02-11T09:50:49.787540619Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wBCuU48Ox_uSnLpuXpSWi95","ver":1},"status":{"status":"write"}}},{"id":"1_1613037049.792438_336.1","section":"bucket","name":"tkscyf-35","timestamp":"2021-02-11T09:50:49.792438114Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wBCuU48Ox_uSnLpuXpSWi95","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037049.814263_337.1","section":"bucket.instance","name":"tkscyf-33:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.33","timestamp":"2021-02-11T09:50:49.814263352Z","data":{"read_version":{"tag":"_eZRAP_8ajxLAWs4hu7nqOwz","ver":1},"write_version":{"tag":"_eZRAP_8ajxLAWs4hu7nqOwz","ver":2},"status":{"status":"write"}}},{"id":"1_1613037049.819262_338.1","section":"bucket.instance","name":"tkscyf-33:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.33","timestamp":"2021-02-11T09:50:49.819262498Z","data":{"read_version":{"tag":"_eZRAP_8ajxLAWs4hu7nqOwz","ver":1},"write_version":{"tag":"_eZRAP_8ajxLAWs4hu7nqOwz","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037050.867169_339.1","section":"bucket.instance","name":"tkscyf-34:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.34","timestamp":"2021-02-11T09:50:50.867169930Z","data":{"read_version":{"tag":"_k7kM2SM39DqWgu3NfoI4bqF","ver":1},"write_version":{"tag":"_k7kM2SM39DqWgu3NfoI4bqF","ver":2},"status":{"status":"write"}}},{"id":"1_1613037050.879211_340.1","section":"bucket.instance","name":"tkscyf-34:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.34","timestamp":"2021-02-11T09:50:50.879211406Z","data":{"read_version":{"tag":"_k7kM2SM39DqWgu3NfoI4bqF","ver":1},"write_version":{"tag":"_k7kM2SM39DqWgu3NfoI4bqF","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037051.935225_341.1","section":"bucket.instance","name":"tkscyf-35:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.35","timestamp":"2021-02-11T09:50:51.935225906Z","data":{"read_version":{"tag":"_KosuwqaTc-yYnhDuNK50-jc","ver":1},"write_version":{"tag":"_KosuwqaTc-yYnhDuNK50-jc","ver":2},"status":{"status":"write"}}},{"id":"1_1613037051.946001_342.1","section":"bucket.instance","name":"tkscyf-35:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.35","timestamp":"2021-02-11T09:50:51.946001361Z","data":{"read_version":{"tag":"_KosuwqaTc-yYnhDuNK50-jc","ver":1},"write_version":{"tag":"_KosuwqaTc-yYnhDuNK50-jc","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037056.396292_343.1","section":"bucket.instance","name":"tkscyf-36:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.36","timestamp":"2021-02-11T09:50:56.396292831Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ssZsgWPzFUlbtHH72rboNt2","ver":1},"status":{"status":"write"}}},{"id":"1_1613037056.402315_344.1","section":"bucket.instance","name":"tkscyf-36:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.36","timestamp":"2021-02-11T09:50:56.402315663Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ssZsgWPzFUlbtHH72rboNt2","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037056.404168_345.1","section":"bucket","name":"tkscyf-36","timestamp":"2021-02-11T09:50:56.404168283Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_R9PAwh1OvxD1UjEPuuj8qio","ver":1},"status":{"status":"write"}}},{"id":"1_1613037056.408811_346.1","section":"bucket","name":"tkscyf-36","timestamp":"2021-02-11T09:50:56.408811301Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_R9PAwh1OvxD1UjEPuuj8qio","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037056.437739_347.1","section":"bucket.instance","name":"tkscyf-37:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.37","timestamp":"2021-02-11T09:50:56.437739913Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HWaNKKBo9OCATJepv1PpQ9b","ver":1},"status":{"status":"write"}}},{"id":"1_1613037056.442573_348.1","section":"bucket.instance","name":"tkscyf-37:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.37","timestamp":"2021-02-11T09:50:56.442573229Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HWaNKKBo9OCATJepv1PpQ9b","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037056.444396_349.1","section":"bucket","name":"tkscyf-37","timestamp":"2021-02-11T09:50:56.444396764Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RbeSoBdgaTvi80yVezMNU9a","ver":1},"status":{"status":"write"}}},{"id":"1_1613037056.449127_350.1","section":"bucket","name":"tkscyf-37","timestamp":"2021-02-11T09:50:56.449127942Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RbeSoBdgaTvi80yVezMNU9a","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037056.503850_351.1","section":"bucket.instance","name":"tkscyf-38:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.38","timestamp":"2021-02-11T09:50:56.503850524Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_M7gYPbaihLb1WY3hhZTdAtg","ver":1},"status":{"status":"write"}}},{"id":"1_1613037056.508532_352.1","section":"bucket.instance","name":"tkscyf-38:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.38","timestamp":"2021-02-11T09:50:56.508532698Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_M7gYPbaihLb1WY3hhZTdAtg","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037056.510275_353.1","section":"bucket","name":"tkscyf-38","timestamp":"2021-02-11T09:50:56.510275102Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0RzXHcbVo20pBILfHDUWraY","ver":1},"status":{"status":"write"}}},{"id":"1_1613037056.514482_354.1","section":"bucket","name":"tkscyf-38","timestamp":"2021-02-11T09:50:56.514482141Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0RzXHcbVo20pBILfHDUWraY","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037057.908443_355.1","section":"bucket.instance","name":"tkscyf-39:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.39","timestamp":"2021-02-11T09:50:57.908443576Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_94HMkmvhzlZXLQvruAOVtAJ","ver":1},"status":{"status":"write"}}},{"id":"1_1613037057.914397_356.1","section":"bucket.instance","name":"tkscyf-39:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.39","timestamp":"2021-02-11T09:50:57.914397041Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_94HMkmvhzlZXLQvruAOVtAJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037057.916715_357.1","section":"bucket","name":"tkscyf-39","timestamp":"2021-02-11T09:50:57.916715595Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wsKJ1QOckGXWesrqOCyo1WG","ver":1},"status":{"status":"write"}}},{"id":"1_1613037057.923356_358.1","section":"bucket","name":"tkscyf-39","timestamp":"2021-02-11T09:50:57.923356621Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wsKJ1QOckGXWesrqOCyo1WG","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037057.945478_359.1","section":"bucket.instance","name":"tkscyf-40:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.40","timestamp":"2021-02-11T09:50:57.945478017Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pg3ahh-4pO-_3XyceN-gWsQ","ver":1},"status":{"status":"write"}}},{"id":"1_1613037057.949753_360.1","section":"bucket.instance","name":"tkscyf-40:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.40","timestamp":"2021-02-11T09:50:57.949753049Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pg3ahh-4pO-_3XyceN-gWsQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037057.951444_361.1","section":"bucket","name":"tkscyf-40","timestamp":"2021-02-11T09:50:57.951444977Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ej1h_jFuYjzyoRvb-1C7RAf","ver":1},"status":{"status":"write"}}},{"id":"1_1613037057.955584_362.1","section":"bucket","name":"tkscyf-40","timestamp":"2021-02-11T09:50:57.955584939Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ej1h_jFuYjzyoRvb-1C7RAf","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037057.996014_363.1","section":"bucket.instance","name":"tkscyf-41:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.41","timestamp":"2021-02-11T09:50:57.996014692Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_I9DAcVu7OInplRV1Ta3wruY","ver":1},"status":{"status":"write"}}},{"id":"1_1613037058.000113_364.1","section":"bucket.instance","name":"tkscyf-41:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.41","timestamp":"2021-02-11T09:50:58.000113498Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_I9DAcVu7OInplRV1Ta3wruY","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037058.001757_365.1","section":"bucket","name":"tkscyf-41","timestamp":"2021-02-11T09:50:58.001757095Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_R0j0lRoRH3ZszkT7rysin2m","ver":1},"status":{"status":"write"}}},{"id":"1_1613037058.005761_366.1","section":"bucket","name":"tkscyf-41","timestamp":"2021-02-11T09:50:58.005761416Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_R0j0lRoRH3ZszkT7rysin2m","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036843.308060_40.1","section":"bucket.instance","name":"tkscyf-1:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.1","timestamp":"2021-02-11T09:47:23.308060874Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vJcQgK-3XDRzzGYAO1_aHfA","ver":1},"status":{"status":"write"}}},{"id":"1_1613036843.334677_41.1","section":"bucket.instance","name":"tkscyf-1:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.1","timestamp":"2021-02-11T09:47:23.334677253Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vJcQgK-3XDRzzGYAO1_aHfA","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036843.341061_42.1","section":"bucket","name":"tkscyf-1","timestamp":"2021-02-11T09:47:23.341061810Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_so1GMpfrEZgmi_sTYZSAr8H","ver":1},"status":{"status":"write"}}},{"id":"1_1613036843.345437_43.1","section":"bucket","name":"tkscyf-1","timestamp":"2021-02-11T09:47:23.345437347Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_so1GMpfrEZgmi_sTYZSAr8H","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036843.366232_44.1","section":"bucket.instance","name":"tkscyf-2:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.2","timestamp":"2021-02-11T09:47:23.366232598Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MZ24WdPDpXrrqQPnafXRqiN","ver":1},"status":{"status":"write"}}},{"id":"1_1613036843.371311_45.1","section":"bucket.instance","name":"tkscyf-2:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.2","timestamp":"2021-02-11T09:47:23.371311720Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MZ24WdPDpXrrqQPnafXRqiN","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036843.373088_46.1","section":"bucket","name":"tkscyf-2","timestamp":"2021-02-11T09:47:23.373088017Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5VYuSj_-KrHdRO5ccn8Xe_y","ver":1},"status":{"status":"write"}}},{"id":"1_1613036843.378050_47.1","section":"bucket","name":"tkscyf-2","timestamp":"2021-02-11T09:47:23.378050996Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5VYuSj_-KrHdRO5ccn8Xe_y","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036847.401028_48.1","section":"bucket.instance","name":"tkscyf-3:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.3","timestamp":"2021-02-11T09:47:27.401028055Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ymULWMaifA40PYB81mtiqKO","ver":1},"status":{"status":"write"}}},{"id":"1_1613036847.406630_49.1","section":"bucket.instance","name":"tkscyf-3:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.3","timestamp":"2021-02-11T09:47:27.406630738Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ymULWMaifA40PYB81mtiqKO","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036847.408599_50.1","section":"bucket","name":"tkscyf-3","timestamp":"2021-02-11T09:47:27.408599295Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8F2ONXee867moT-prh9FMvz","ver":1},"status":{"status":"write"}}},{"id":"1_1613036847.414041_51.1","section":"bucket","name":"tkscyf-3","timestamp":"2021-02-11T09:47:27.414041891Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8F2ONXee867moT-prh9FMvz","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036848.515900_52.1","section":"bucket.instance","name":"tkscyf-4:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.4","timestamp":"2021-02-11T09:47:28.515900958Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OC0h4nNzRptiVogF5DDSlbl","ver":1},"status":{"status":"write"}}},{"id":"1_1613036848.521086_53.1","section":"bucket.instance","name":"tkscyf-4:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.4","timestamp":"2021-02-11T09:47:28.521086371Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OC0h4nNzRptiVogF5DDSlbl","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036848.523142_54.1","section":"bucket","name":"tkscyf-4","timestamp":"2021-02-11T09:47:28.523142343Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lzBpaQfMhBTasrckHQnxsrH","ver":1},"status":{"status":"write"}}},{"id":"1_1613036848.528085_55.1","section":"bucket","name":"tkscyf-4","timestamp":"2021-02-11T09:47:28.528085857Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lzBpaQfMhBTasrckHQnxsrH","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036848.545712_56.1","section":"bucket.instance","name":"tkscyf-5:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.5","timestamp":"2021-02-11T09:47:28.545712519Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pizM_4Vxb6ga5d7zQm5cup4","ver":1},"status":{"status":"write"}}},{"id":"1_1613036848.551316_57.1","section":"bucket.instance","name":"tkscyf-5:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.5","timestamp":"2021-02-11T09:47:28.551316982Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pizM_4Vxb6ga5d7zQm5cup4","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036848.553108_58.1","section":"bucket","name":"tkscyf-5","timestamp":"2021-02-11T09:47:28.553108370Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__btqQaGnBiVqesmRij4KpZu","ver":1},"status":{"status":"write"}}},{"id":"1_1613036848.557871_59.1","section":"bucket","name":"tkscyf-5","timestamp":"2021-02-11T09:47:28.557871264Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__btqQaGnBiVqesmRij4KpZu","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036848.598426_60.1","section":"bucket.instance","name":"tkscyf-6:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.6","timestamp":"2021-02-11T09:47:28.598426655Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MXFYvcf4zPQLqC1AyVkpK27","ver":1},"status":{"status":"write"}}},{"id":"1_1613036848.607442_61.1","section":"bucket.instance","name":"tkscyf-6:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.6","timestamp":"2021-02-11T09:47:28.607442762Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MXFYvcf4zPQLqC1AyVkpK27","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036848.609097_62.1","section":"bucket","name":"tkscyf-6","timestamp":"2021-02-11T09:47:28.609097002Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_27D4N2MBNgnn-CUkGMJjydq","ver":1},"status":{"status":"write"}}},{"id":"1_1613036848.613340_63.1","section":"bucket","name":"tkscyf-6","timestamp":"2021-02-11T09:47:28.613340684Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_27D4N2MBNgnn-CUkGMJjydq","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036851.051651_64.1","section":"bucket.instance","name":"tkscyf-7:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.7","timestamp":"2021-02-11T09:47:31.051651058Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qdftsvlxuD5g0Vn1ajzNmyo","ver":1},"status":{"status":"write"}}},{"id":"1_1613036851.056767_65.1","section":"bucket.instance","name":"tkscyf-7:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.7","timestamp":"2021-02-11T09:47:31.056767922Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qdftsvlxuD5g0Vn1ajzNmyo","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036851.058594_66.1","section":"bucket","name":"tkscyf-7","timestamp":"2021-02-11T09:47:31.058594967Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FCpy5wrdCKP58fp9iJu68WQ","ver":1},"status":{"status":"write"}}},{"id":"1_1613036851.063396_67.1","section":"bucket","name":"tkscyf-7","timestamp":"2021-02-11T09:47:31.063396372Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FCpy5wrdCKP58fp9iJu68WQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036851.082324_68.1","section":"bucket.instance","name":"tkscyf-8:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.8","timestamp":"2021-02-11T09:47:31.082324615Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZgPd4GGXu--UcP5ytH49TD4","ver":1},"status":{"status":"write"}}},{"id":"1_1613036851.087320_69.1","section":"bucket.instance","name":"tkscyf-8:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.8","timestamp":"2021-02-11T09:47:31.087320996Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZgPd4GGXu--UcP5ytH49TD4","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036851.089023_70.1","section":"bucket","name":"tkscyf-8","timestamp":"2021-02-11T09:47:31.089023916Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MfurQNSSeZct4L9MZ9BG8Gr","ver":1},"status":{"status":"write"}}},{"id":"1_1613036851.093557_71.1","section":"bucket","name":"tkscyf-8","timestamp":"2021-02-11T09:47:31.093557563Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MfurQNSSeZct4L9MZ9BG8Gr","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036851.133487_72.1","section":"bucket.instance","name":"tkscyf-9:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.9","timestamp":"2021-02-11T09:47:31.133487717Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gym2dhV8yR1PgdklxT3K2xe","ver":1},"status":{"status":"write"}}},{"id":"1_1613036851.138269_73.1","section":"bucket.instance","name":"tkscyf-9:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.9","timestamp":"2021-02-11T09:47:31.138269287Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gym2dhV8yR1PgdklxT3K2xe","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036851.139876_74.1","section":"bucket","name":"tkscyf-9","timestamp":"2021-02-11T09:47:31.139876187Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_p0bbqr2Y572m56xpGPFsiZ-","ver":1},"status":{"status":"write"}}},{"id":"1_1613036851.144668_75.1","section":"bucket","name":"tkscyf-9","timestamp":"2021-02-11T09:47:31.144668547Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_p0bbqr2Y572m56xpGPFsiZ-","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036863.874762_76.1","section":"bucket","name":"tkscyf-7","timestamp":"2021-02-11T09:47:43.874762601Z","data":{"read_version":{"tag":"_FCpy5wrdCKP58fp9iJu68WQ","ver":1},"write_version":{"tag":"_FCpy5wrdCKP58fp9iJu68WQ","ver":2},"status":{"status":"remove"}}},{"id":"1_1613036863.880950_77.1","section":"bucket","name":"tkscyf-7","timestamp":"2021-02-11T09:47:43.880950937Z","data":{"read_version":{"tag":"_FCpy5wrdCKP58fp9iJu68WQ","ver":1},"write_version":{"tag":"_FCpy5wrdCKP58fp9iJu68WQ","ver":2},"status":{"status":"complete"}}},{"id":"1_1613036863.956967_78.1","section":"bucket","name":"tkscyf-8","timestamp":"2021-02-11T09:47:43.956967900Z","data":{"read_version":{"tag":"_MfurQNSSeZct4L9MZ9BG8Gr","ver":1},"write_version":{"tag":"_MfurQNSSeZct4L9MZ9BG8Gr","ver":2},"status":{"status":"remove"}}},{"id":"1_1613036863.963661_79.1","section":"bucket","name":"tkscyf-8","timestamp":"2021-02-11T09:47:43.963661824Z","data":{"read_version":{"tag":"_MfurQNSSeZct4L9MZ9BG8Gr","ver":1},"write_version":{"tag":"_MfurQNSSeZct4L9MZ9BG8Gr","ver":2},"status":{"status":"complete"}}},{"id":"1_1613036864.074054_80.1","section":"bucket","name":"tkscyf-9","timestamp":"2021-02-11T09:47:44.074054305Z","data":{"read_version":{"tag":"_p0bbqr2Y572m56xpGPFsiZ-","ver":1},"write_version":{"tag":"_p0bbqr2Y572m56xpGPFsiZ-","ver":2},"status":{"status":"remove"}}},{"id":"1_1613036864.081584_81.1","section":"bucket","name":"tkscyf-9","timestamp":"2021-02-11T09:47:44.081584235Z","data":{"read_version":{"tag":"_p0bbqr2Y572m56xpGPFsiZ-","ver":1},"write_version":{"tag":"_p0bbqr2Y572m56xpGPFsiZ-","ver":2},"status":{"status":"complete"}}}] 2021-02-11T09:53:15.041 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:15.041 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:15.041 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:15.079 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:15.079 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:15.333 INFO:teuthology.orchestra.run.smithi098.stdout:[{"id":"1_1613036708.088554_48.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2021-02-11T09:45:08.088554566Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RA3OvQQVI2OcioTs6vG61ts","ver":1},"status":{"status":"write"}}},{"id":"1_1613036708.096437_52.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2021-02-11T09:45:08.096437057Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RA3OvQQVI2OcioTs6vG61ts","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036865.186535_60.1","section":"bucket.instance","name":"tkscyf-10:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.10","timestamp":"2021-02-11T09:47:45.186535085Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BOQUev-Msz6Gg8HQx3I-Uw_","ver":1},"status":{"status":"write"}}},{"id":"1_1613036865.191608_62.1","section":"bucket.instance","name":"tkscyf-10:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.10","timestamp":"2021-02-11T09:47:45.191608247Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BOQUev-Msz6Gg8HQx3I-Uw_","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036865.193557_63.1","section":"bucket","name":"tkscyf-10","timestamp":"2021-02-11T09:47:45.193557933Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fhFnaBfHBt4OCc3A0Cy3U0l","ver":1},"status":{"status":"write"}}},{"id":"1_1613036865.198089_64.1","section":"bucket","name":"tkscyf-10","timestamp":"2021-02-11T09:47:45.198089937Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fhFnaBfHBt4OCc3A0Cy3U0l","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036865.217116_66.1","section":"bucket.instance","name":"tkscyf-11:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.11","timestamp":"2021-02-11T09:47:45.217116956Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_x_UhJIyltXoCF5sGbvGrvWs","ver":1},"status":{"status":"write"}}},{"id":"1_1613036865.222221_67.1","section":"bucket.instance","name":"tkscyf-11:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.11","timestamp":"2021-02-11T09:47:45.222221108Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_x_UhJIyltXoCF5sGbvGrvWs","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036865.224285_68.1","section":"bucket","name":"tkscyf-11","timestamp":"2021-02-11T09:47:45.224285083Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FRR9uzZkCcSEP86_Kp419wb","ver":1},"status":{"status":"write"}}},{"id":"1_1613036865.228963_69.1","section":"bucket","name":"tkscyf-11","timestamp":"2021-02-11T09:47:45.228963299Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FRR9uzZkCcSEP86_Kp419wb","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036865.267357_70.1","section":"bucket.instance","name":"tkscyf-12:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.12","timestamp":"2021-02-11T09:47:45.267357171Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QtLjmo_4M7YVwSusOVuvKW7","ver":1},"status":{"status":"write"}}},{"id":"1_1613036865.272260_71.1","section":"bucket.instance","name":"tkscyf-12:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.12","timestamp":"2021-02-11T09:47:45.272260535Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QtLjmo_4M7YVwSusOVuvKW7","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036865.273997_72.1","section":"bucket","name":"tkscyf-12","timestamp":"2021-02-11T09:47:45.273997074Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9ZVFThqlMn2Lww6c1ZLx1-O","ver":1},"status":{"status":"write"}}},{"id":"1_1613036865.278468_73.1","section":"bucket","name":"tkscyf-12","timestamp":"2021-02-11T09:47:45.278468126Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9ZVFThqlMn2Lww6c1ZLx1-O","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036885.857139_178.1","section":"bucket.instance","name":"tkscyf-13:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.13","timestamp":"2021-02-11T09:48:05.857139297Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LjOnUQz6uMNf3yxGrmeonuy","ver":1},"status":{"status":"write"}}},{"id":"1_1613036885.861989_179.1","section":"bucket.instance","name":"tkscyf-13:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.13","timestamp":"2021-02-11T09:48:05.861989993Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LjOnUQz6uMNf3yxGrmeonuy","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036885.863884_180.1","section":"bucket","name":"tkscyf-13","timestamp":"2021-02-11T09:48:05.863884773Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f0vlqtQxcyVJgs3vWIVlMY5","ver":1},"status":{"status":"write"}}},{"id":"1_1613036885.868524_181.1","section":"bucket","name":"tkscyf-13","timestamp":"2021-02-11T09:48:05.868524702Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f0vlqtQxcyVJgs3vWIVlMY5","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036885.887239_182.1","section":"bucket.instance","name":"tkscyf-14:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.14","timestamp":"2021-02-11T09:48:05.887239288Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Cb1zieCDGGuK7mRxM9X7cOY","ver":1},"status":{"status":"write"}}},{"id":"1_1613036885.892744_183.1","section":"bucket.instance","name":"tkscyf-14:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.14","timestamp":"2021-02-11T09:48:05.892744502Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Cb1zieCDGGuK7mRxM9X7cOY","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036885.894666_184.1","section":"bucket","name":"tkscyf-14","timestamp":"2021-02-11T09:48:05.894666449Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0d8lwgEigMOkq-FSumkqW71","ver":1},"status":{"status":"write"}}},{"id":"1_1613036885.900682_185.1","section":"bucket","name":"tkscyf-14","timestamp":"2021-02-11T09:48:05.900682555Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0d8lwgEigMOkq-FSumkqW71","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036885.942706_186.1","section":"bucket.instance","name":"tkscyf-15:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.15","timestamp":"2021-02-11T09:48:05.942706611Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-hfQZ5lWiy4o6NP7OEZNQ0h","ver":1},"status":{"status":"write"}}},{"id":"1_1613036885.947464_187.1","section":"bucket.instance","name":"tkscyf-15:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.15","timestamp":"2021-02-11T09:48:05.947464351Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-hfQZ5lWiy4o6NP7OEZNQ0h","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036885.949248_188.1","section":"bucket","name":"tkscyf-15","timestamp":"2021-02-11T09:48:05.949248580Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4gdcmHzvr3iqrlpLrZQpyg1","ver":1},"status":{"status":"write"}}},{"id":"1_1613036885.953989_189.1","section":"bucket","name":"tkscyf-15","timestamp":"2021-02-11T09:48:05.953989244Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4gdcmHzvr3iqrlpLrZQpyg1","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036899.854288_194.1","section":"bucket.instance","name":"tkscyf-16:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.16","timestamp":"2021-02-11T09:48:19.854288478Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BjHGVIhMvUslBcZQfe5MTso","ver":1},"status":{"status":"write"}}},{"id":"1_1613036899.859377_195.1","section":"bucket.instance","name":"tkscyf-16:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.16","timestamp":"2021-02-11T09:48:19.859377386Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BjHGVIhMvUslBcZQfe5MTso","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036899.861235_196.1","section":"bucket","name":"tkscyf-16","timestamp":"2021-02-11T09:48:19.861235489Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_mBUX065ZmzTTf0GD1alWPkY","ver":1},"status":{"status":"write"}}},{"id":"1_1613036899.866335_197.1","section":"bucket","name":"tkscyf-16","timestamp":"2021-02-11T09:48:19.866335427Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_mBUX065ZmzTTf0GD1alWPkY","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036899.888709_198.1","section":"bucket.instance","name":"tkscyf-17:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.17","timestamp":"2021-02-11T09:48:19.888709670Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_upQr8yx99-au6_AWkKxYuHu","ver":1},"status":{"status":"write"}}},{"id":"1_1613036899.894272_199.1","section":"bucket.instance","name":"tkscyf-17:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.17","timestamp":"2021-02-11T09:48:19.894272943Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_upQr8yx99-au6_AWkKxYuHu","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036899.896430_200.1","section":"bucket","name":"tkscyf-17","timestamp":"2021-02-11T09:48:19.896430934Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_EVefVFc5TujdFr9n6vuddup","ver":1},"status":{"status":"write"}}},{"id":"1_1613036899.901399_201.1","section":"bucket","name":"tkscyf-17","timestamp":"2021-02-11T09:48:19.901399970Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_EVefVFc5TujdFr9n6vuddup","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036899.942394_202.1","section":"bucket.instance","name":"tkscyf-18:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.18","timestamp":"2021-02-11T09:48:19.942394398Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eLelEHOSgz_5RWVJgejlcwE","ver":1},"status":{"status":"write"}}},{"id":"1_1613036899.947586_203.1","section":"bucket.instance","name":"tkscyf-18:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.18","timestamp":"2021-02-11T09:48:19.947586263Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eLelEHOSgz_5RWVJgejlcwE","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036899.949797_204.1","section":"bucket","name":"tkscyf-18","timestamp":"2021-02-11T09:48:19.949797527Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nnXNYQhXpslLZMkcMsbaLYW","ver":1},"status":{"status":"write"}}},{"id":"1_1613036899.954577_205.1","section":"bucket","name":"tkscyf-18","timestamp":"2021-02-11T09:48:19.954577937Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nnXNYQhXpslLZMkcMsbaLYW","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036899.976313_206.1","section":"bucket.instance","name":"tkscyf-16:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.16","timestamp":"2021-02-11T09:48:19.976313350Z","data":{"read_version":{"tag":"_BjHGVIhMvUslBcZQfe5MTso","ver":1},"write_version":{"tag":"_BjHGVIhMvUslBcZQfe5MTso","ver":2},"status":{"status":"write"}}},{"id":"1_1613036899.981307_207.1","section":"bucket.instance","name":"tkscyf-16:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.16","timestamp":"2021-02-11T09:48:19.981307840Z","data":{"read_version":{"tag":"_BjHGVIhMvUslBcZQfe5MTso","ver":1},"write_version":{"tag":"_BjHGVIhMvUslBcZQfe5MTso","ver":2},"status":{"status":"complete"}}},{"id":"1_1613036899.986228_208.1","section":"bucket.instance","name":"tkscyf-17:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.17","timestamp":"2021-02-11T09:48:19.986228789Z","data":{"read_version":{"tag":"_upQr8yx99-au6_AWkKxYuHu","ver":1},"write_version":{"tag":"_upQr8yx99-au6_AWkKxYuHu","ver":2},"status":{"status":"write"}}},{"id":"1_1613036899.991766_209.1","section":"bucket.instance","name":"tkscyf-17:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.17","timestamp":"2021-02-11T09:48:19.991766410Z","data":{"read_version":{"tag":"_upQr8yx99-au6_AWkKxYuHu","ver":1},"write_version":{"tag":"_upQr8yx99-au6_AWkKxYuHu","ver":2},"status":{"status":"complete"}}},{"id":"1_1613036900.005531_210.1","section":"bucket.instance","name":"tkscyf-18:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.18","timestamp":"2021-02-11T09:48:20.005531886Z","data":{"read_version":{"tag":"_eLelEHOSgz_5RWVJgejlcwE","ver":1},"write_version":{"tag":"_eLelEHOSgz_5RWVJgejlcwE","ver":2},"status":{"status":"write"}}},{"id":"1_1613036900.015316_211.1","section":"bucket.instance","name":"tkscyf-18:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.18","timestamp":"2021-02-11T09:48:20.015316565Z","data":{"read_version":{"tag":"_eLelEHOSgz_5RWVJgejlcwE","ver":1},"write_version":{"tag":"_eLelEHOSgz_5RWVJgejlcwE","ver":2},"status":{"status":"complete"}}},{"id":"1_1613036950.507710_227.1","section":"bucket.instance","name":"tkscyf-19:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.19","timestamp":"2021-02-11T09:49:10.507710607Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_G8JIgZ6gi-tWTxNuGHdxLfV","ver":1},"status":{"status":"write"}}},{"id":"1_1613036950.512728_228.1","section":"bucket.instance","name":"tkscyf-19:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.19","timestamp":"2021-02-11T09:49:10.512728298Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_G8JIgZ6gi-tWTxNuGHdxLfV","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036950.514564_229.1","section":"bucket","name":"tkscyf-19","timestamp":"2021-02-11T09:49:10.514564712Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rUH9m8tDhvujVPRfyQMubOa","ver":1},"status":{"status":"write"}}},{"id":"1_1613036950.519470_230.1","section":"bucket","name":"tkscyf-19","timestamp":"2021-02-11T09:49:10.519470166Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rUH9m8tDhvujVPRfyQMubOa","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036950.525237_231.1","section":"bucket.instance","name":"tkscyf-19:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.19","timestamp":"2021-02-11T09:49:10.525237169Z","data":{"read_version":{"tag":"_G8JIgZ6gi-tWTxNuGHdxLfV","ver":1},"write_version":{"tag":"_G8JIgZ6gi-tWTxNuGHdxLfV","ver":2},"status":{"status":"write"}}},{"id":"1_1613036950.530761_232.1","section":"bucket.instance","name":"tkscyf-19:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.19","timestamp":"2021-02-11T09:49:10.530761661Z","data":{"read_version":{"tag":"_G8JIgZ6gi-tWTxNuGHdxLfV","ver":1},"write_version":{"tag":"_G8JIgZ6gi-tWTxNuGHdxLfV","ver":2},"status":{"status":"complete"}}},{"id":"1_1613036998.267812_238.1","section":"bucket.instance","name":"tkscyf-20:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.20","timestamp":"2021-02-11T09:49:58.267812499Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3X25hzqCxgrngeWBjXAq3MW","ver":1},"status":{"status":"write"}}},{"id":"1_1613036998.273260_239.1","section":"bucket.instance","name":"tkscyf-20:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.20","timestamp":"2021-02-11T09:49:58.273260665Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3X25hzqCxgrngeWBjXAq3MW","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036998.275321_240.1","section":"bucket","name":"tkscyf-20","timestamp":"2021-02-11T09:49:58.275321795Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GSd-2J0lHNWhUfK9gtbzsaz","ver":1},"status":{"status":"write"}}},{"id":"1_1613036998.280111_241.1","section":"bucket","name":"tkscyf-20","timestamp":"2021-02-11T09:49:58.280111713Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GSd-2J0lHNWhUfK9gtbzsaz","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037007.140764_243.1","section":"bucket.instance","name":"tkscyf-20:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.20","timestamp":"2021-02-11T09:50:07.140764026Z","data":{"read_version":{"tag":"_3X25hzqCxgrngeWBjXAq3MW","ver":1},"write_version":{"tag":"_3X25hzqCxgrngeWBjXAq3MW","ver":2},"status":{"status":"write"}}},{"id":"1_1613037007.150589_244.1","section":"bucket.instance","name":"tkscyf-20:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.20","timestamp":"2021-02-11T09:50:07.150589879Z","data":{"read_version":{"tag":"_3X25hzqCxgrngeWBjXAq3MW","ver":1},"write_version":{"tag":"_3X25hzqCxgrngeWBjXAq3MW","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037012.304895_245.1","section":"bucket.instance","name":"tkscyf-20:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.20","timestamp":"2021-02-11T09:50:12.304895189Z","data":{"read_version":{"tag":"_3X25hzqCxgrngeWBjXAq3MW","ver":2},"write_version":{"tag":"_3X25hzqCxgrngeWBjXAq3MW","ver":3},"status":{"status":"write"}}},{"id":"1_1613037012.316265_246.1","section":"bucket.instance","name":"tkscyf-20:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.20","timestamp":"2021-02-11T09:50:12.316265099Z","data":{"read_version":{"tag":"_3X25hzqCxgrngeWBjXAq3MW","ver":2},"write_version":{"tag":"_3X25hzqCxgrngeWBjXAq3MW","ver":3},"status":{"status":"complete"}}},{"id":"1_1613037016.767451_247.1","section":"bucket.instance","name":"tkscyf-21:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.21","timestamp":"2021-02-11T09:50:16.767451636Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_usNhex5LVJs2K5fhCTS-E9B","ver":1},"status":{"status":"write"}}},{"id":"1_1613037016.772791_248.1","section":"bucket.instance","name":"tkscyf-21:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.21","timestamp":"2021-02-11T09:50:16.772791796Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_usNhex5LVJs2K5fhCTS-E9B","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037016.774734_249.1","section":"bucket","name":"tkscyf-21","timestamp":"2021-02-11T09:50:16.774734911Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_oS8fsXNPnQ3YSGsw6yEPI6c","ver":1},"status":{"status":"write"}}},{"id":"1_1613037016.779555_250.1","section":"bucket","name":"tkscyf-21","timestamp":"2021-02-11T09:50:16.779555063Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_oS8fsXNPnQ3YSGsw6yEPI6c","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037016.800629_251.1","section":"bucket.instance","name":"tkscyf-22:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.22","timestamp":"2021-02-11T09:50:16.800629460Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Q5XnU57eMsSQj5AxurjrM1s","ver":1},"status":{"status":"write"}}},{"id":"1_1613037016.807098_252.1","section":"bucket.instance","name":"tkscyf-22:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.22","timestamp":"2021-02-11T09:50:16.807098120Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Q5XnU57eMsSQj5AxurjrM1s","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037016.809071_253.1","section":"bucket","name":"tkscyf-22","timestamp":"2021-02-11T09:50:16.809071480Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Jcw-M7gbUwOe2t495PGK9vo","ver":1},"status":{"status":"write"}}},{"id":"1_1613037016.813672_254.1","section":"bucket","name":"tkscyf-22","timestamp":"2021-02-11T09:50:16.813672741Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Jcw-M7gbUwOe2t495PGK9vo","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037016.860072_255.1","section":"bucket.instance","name":"tkscyf-23:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.23","timestamp":"2021-02-11T09:50:16.860072689Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GCLr8VFsBOKFg5ZuZUx_GvZ","ver":1},"status":{"status":"write"}}},{"id":"1_1613037016.866566_256.1","section":"bucket.instance","name":"tkscyf-23:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.23","timestamp":"2021-02-11T09:50:16.866566184Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GCLr8VFsBOKFg5ZuZUx_GvZ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037016.869661_257.1","section":"bucket","name":"tkscyf-23","timestamp":"2021-02-11T09:50:16.869661986Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8HFWSnbWVi7UHPWj_xF4vXf","ver":1},"status":{"status":"write"}}},{"id":"1_1613037016.879001_258.1","section":"bucket","name":"tkscyf-23","timestamp":"2021-02-11T09:50:16.879001731Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8HFWSnbWVi7UHPWj_xF4vXf","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037016.897902_259.1","section":"bucket.instance","name":"tkscyf-21:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.21","timestamp":"2021-02-11T09:50:16.897902097Z","data":{"read_version":{"tag":"_usNhex5LVJs2K5fhCTS-E9B","ver":1},"write_version":{"tag":"_usNhex5LVJs2K5fhCTS-E9B","ver":2},"status":{"status":"write"}}},{"id":"1_1613037016.902295_260.1","section":"bucket.instance","name":"tkscyf-21:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.21","timestamp":"2021-02-11T09:50:16.902295553Z","data":{"read_version":{"tag":"_usNhex5LVJs2K5fhCTS-E9B","ver":1},"write_version":{"tag":"_usNhex5LVJs2K5fhCTS-E9B","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037016.906251_261.1","section":"bucket.instance","name":"tkscyf-22:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.22","timestamp":"2021-02-11T09:50:16.906251476Z","data":{"read_version":{"tag":"_Q5XnU57eMsSQj5AxurjrM1s","ver":1},"write_version":{"tag":"_Q5XnU57eMsSQj5AxurjrM1s","ver":2},"status":{"status":"write"}}},{"id":"1_1613037016.911598_262.1","section":"bucket.instance","name":"tkscyf-22:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.22","timestamp":"2021-02-11T09:50:16.911598777Z","data":{"read_version":{"tag":"_Q5XnU57eMsSQj5AxurjrM1s","ver":1},"write_version":{"tag":"_Q5XnU57eMsSQj5AxurjrM1s","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037016.926597_263.1","section":"bucket.instance","name":"tkscyf-23:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.23","timestamp":"2021-02-11T09:50:16.926597890Z","data":{"read_version":{"tag":"_GCLr8VFsBOKFg5ZuZUx_GvZ","ver":1},"write_version":{"tag":"_GCLr8VFsBOKFg5ZuZUx_GvZ","ver":2},"status":{"status":"write"}}},{"id":"1_1613037016.931312_264.1","section":"bucket.instance","name":"tkscyf-23:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.23","timestamp":"2021-02-11T09:50:16.931312694Z","data":{"read_version":{"tag":"_GCLr8VFsBOKFg5ZuZUx_GvZ","ver":1},"write_version":{"tag":"_GCLr8VFsBOKFg5ZuZUx_GvZ","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037035.238424_265.1","section":"bucket.instance","name":"tkscyf-24:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.24","timestamp":"2021-02-11T09:50:35.238424865Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bb0kUiSwNRO-U3TZ4E1z9m5","ver":1},"status":{"status":"write"}}},{"id":"1_1613037035.243640_266.1","section":"bucket.instance","name":"tkscyf-24:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.24","timestamp":"2021-02-11T09:50:35.243640241Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bb0kUiSwNRO-U3TZ4E1z9m5","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037035.245475_267.1","section":"bucket","name":"tkscyf-24","timestamp":"2021-02-11T09:50:35.245475848Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7Yg_eLnx93F7TXgKcKQAxT5","ver":1},"status":{"status":"write"}}},{"id":"1_1613037035.250290_268.1","section":"bucket","name":"tkscyf-24","timestamp":"2021-02-11T09:50:35.250290672Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7Yg_eLnx93F7TXgKcKQAxT5","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037035.274261_269.1","section":"bucket.instance","name":"tkscyf-25:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.25","timestamp":"2021-02-11T09:50:35.274261140Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1nnotJE0A2l-djRVSBIWs9E","ver":1},"status":{"status":"write"}}},{"id":"1_1613037035.281072_270.1","section":"bucket.instance","name":"tkscyf-25:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.25","timestamp":"2021-02-11T09:50:35.281072011Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1nnotJE0A2l-djRVSBIWs9E","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037035.283085_271.1","section":"bucket","name":"tkscyf-25","timestamp":"2021-02-11T09:50:35.283085981Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PBWPxzfNM3rpuuSRbw9WWJ4","ver":1},"status":{"status":"write"}}},{"id":"1_1613037035.287331_272.1","section":"bucket","name":"tkscyf-25","timestamp":"2021-02-11T09:50:35.287331936Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PBWPxzfNM3rpuuSRbw9WWJ4","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037035.327494_273.1","section":"bucket.instance","name":"tkscyf-26:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.26","timestamp":"2021-02-11T09:50:35.327494695Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yBW2OnCxamt6jwtfytnSIYp","ver":1},"status":{"status":"write"}}},{"id":"1_1613037035.336740_274.1","section":"bucket.instance","name":"tkscyf-26:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.26","timestamp":"2021-02-11T09:50:35.336740442Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yBW2OnCxamt6jwtfytnSIYp","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037035.338820_275.1","section":"bucket","name":"tkscyf-26","timestamp":"2021-02-11T09:50:35.338820669Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GBS2JL6Meg0eeenzd3LFcRc","ver":1},"status":{"status":"write"}}},{"id":"1_1613037035.343318_276.1","section":"bucket","name":"tkscyf-26","timestamp":"2021-02-11T09:50:35.343318410Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GBS2JL6Meg0eeenzd3LFcRc","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037035.361989_277.1","section":"bucket.instance","name":"tkscyf-24:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.24","timestamp":"2021-02-11T09:50:35.361989484Z","data":{"read_version":{"tag":"_bb0kUiSwNRO-U3TZ4E1z9m5","ver":1},"write_version":{"tag":"_bb0kUiSwNRO-U3TZ4E1z9m5","ver":2},"status":{"status":"write"}}},{"id":"1_1613037035.367759_278.1","section":"bucket.instance","name":"tkscyf-24:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.24","timestamp":"2021-02-11T09:50:35.367759053Z","data":{"read_version":{"tag":"_bb0kUiSwNRO-U3TZ4E1z9m5","ver":1},"write_version":{"tag":"_bb0kUiSwNRO-U3TZ4E1z9m5","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037035.371457_279.1","section":"bucket.instance","name":"tkscyf-24:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.24","timestamp":"2021-02-11T09:50:35.371457040Z","data":{"read_version":{"tag":"_bb0kUiSwNRO-U3TZ4E1z9m5","ver":2},"write_version":{"tag":"_bb0kUiSwNRO-U3TZ4E1z9m5","ver":3},"status":{"status":"write"}}},{"id":"1_1613037035.376968_280.1","section":"bucket.instance","name":"tkscyf-24:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.24","timestamp":"2021-02-11T09:50:35.376968483Z","data":{"read_version":{"tag":"_bb0kUiSwNRO-U3TZ4E1z9m5","ver":2},"write_version":{"tag":"_bb0kUiSwNRO-U3TZ4E1z9m5","ver":3},"status":{"status":"complete"}}},{"id":"1_1613037035.382067_281.1","section":"bucket.instance","name":"tkscyf-25:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.25","timestamp":"2021-02-11T09:50:35.382067509Z","data":{"read_version":{"tag":"_1nnotJE0A2l-djRVSBIWs9E","ver":1},"write_version":{"tag":"_1nnotJE0A2l-djRVSBIWs9E","ver":2},"status":{"status":"write"}}},{"id":"1_1613037035.388230_282.1","section":"bucket.instance","name":"tkscyf-25:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.25","timestamp":"2021-02-11T09:50:35.388230116Z","data":{"read_version":{"tag":"_1nnotJE0A2l-djRVSBIWs9E","ver":1},"write_version":{"tag":"_1nnotJE0A2l-djRVSBIWs9E","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037035.399024_283.1","section":"bucket.instance","name":"tkscyf-25:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.25","timestamp":"2021-02-11T09:50:35.399024419Z","data":{"read_version":{"tag":"_1nnotJE0A2l-djRVSBIWs9E","ver":2},"write_version":{"tag":"_1nnotJE0A2l-djRVSBIWs9E","ver":3},"status":{"status":"write"}}},{"id":"1_1613037035.406913_284.1","section":"bucket.instance","name":"tkscyf-25:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.25","timestamp":"2021-02-11T09:50:35.406913878Z","data":{"read_version":{"tag":"_1nnotJE0A2l-djRVSBIWs9E","ver":2},"write_version":{"tag":"_1nnotJE0A2l-djRVSBIWs9E","ver":3},"status":{"status":"complete"}}},{"id":"1_1613037035.418911_285.1","section":"bucket.instance","name":"tkscyf-26:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.26","timestamp":"2021-02-11T09:50:35.418911493Z","data":{"read_version":{"tag":"_yBW2OnCxamt6jwtfytnSIYp","ver":1},"write_version":{"tag":"_yBW2OnCxamt6jwtfytnSIYp","ver":2},"status":{"status":"write"}}},{"id":"1_1613037035.427825_286.1","section":"bucket.instance","name":"tkscyf-26:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.26","timestamp":"2021-02-11T09:50:35.427825538Z","data":{"read_version":{"tag":"_yBW2OnCxamt6jwtfytnSIYp","ver":1},"write_version":{"tag":"_yBW2OnCxamt6jwtfytnSIYp","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037035.439773_287.1","section":"bucket.instance","name":"tkscyf-26:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.26","timestamp":"2021-02-11T09:50:35.439773122Z","data":{"read_version":{"tag":"_yBW2OnCxamt6jwtfytnSIYp","ver":2},"write_version":{"tag":"_yBW2OnCxamt6jwtfytnSIYp","ver":3},"status":{"status":"write"}}},{"id":"1_1613037035.445782_288.1","section":"bucket.instance","name":"tkscyf-26:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.26","timestamp":"2021-02-11T09:50:35.445782025Z","data":{"read_version":{"tag":"_yBW2OnCxamt6jwtfytnSIYp","ver":2},"write_version":{"tag":"_yBW2OnCxamt6jwtfytnSIYp","ver":3},"status":{"status":"complete"}}},{"id":"1_1613037049.014616_289.1","section":"bucket.instance","name":"tkscyf-27:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.27","timestamp":"2021-02-11T09:50:49.014616484Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HPEAummOtXzD3iTYGsjL78L","ver":1},"status":{"status":"write"}}},{"id":"1_1613037049.020486_290.1","section":"bucket.instance","name":"tkscyf-27:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.27","timestamp":"2021-02-11T09:50:49.020486409Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HPEAummOtXzD3iTYGsjL78L","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037049.022446_291.1","section":"bucket","name":"tkscyf-27","timestamp":"2021-02-11T09:50:49.022446056Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HzrXfJNvB4DsZqVjRjsmkDL","ver":1},"status":{"status":"write"}}},{"id":"1_1613037049.026934_292.1","section":"bucket","name":"tkscyf-27","timestamp":"2021-02-11T09:50:49.026934689Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HzrXfJNvB4DsZqVjRjsmkDL","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037049.052544_293.1","section":"bucket.instance","name":"tkscyf-28:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.28","timestamp":"2021-02-11T09:50:49.052544751Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-VSdKrlRKUfuwloTfMI6Xuv","ver":1},"status":{"status":"write"}}},{"id":"1_1613037049.057739_294.1","section":"bucket.instance","name":"tkscyf-28:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.28","timestamp":"2021-02-11T09:50:49.057739941Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-VSdKrlRKUfuwloTfMI6Xuv","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037049.059973_295.1","section":"bucket","name":"tkscyf-28","timestamp":"2021-02-11T09:50:49.059973620Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-BX9bB1FIeOQc9AK3EsxPMy","ver":1},"status":{"status":"write"}}},{"id":"1_1613037049.064978_296.1","section":"bucket","name":"tkscyf-28","timestamp":"2021-02-11T09:50:49.064978209Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-BX9bB1FIeOQc9AK3EsxPMy","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037049.129163_297.1","section":"bucket.instance","name":"tkscyf-29:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.29","timestamp":"2021-02-11T09:50:49.129163449Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hyJCgZ3ZqhjZ2B82NLQtbFj","ver":1},"status":{"status":"write"}}},{"id":"1_1613037049.139199_298.1","section":"bucket.instance","name":"tkscyf-29:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.29","timestamp":"2021-02-11T09:50:49.139199855Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hyJCgZ3ZqhjZ2B82NLQtbFj","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037049.141005_299.1","section":"bucket","name":"tkscyf-29","timestamp":"2021-02-11T09:50:49.141005489Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1eRc33iNyGdkXUg6pUKi_qn","ver":1},"status":{"status":"write"}}},{"id":"1_1613037049.145434_300.1","section":"bucket","name":"tkscyf-29","timestamp":"2021-02-11T09:50:49.145434314Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1eRc33iNyGdkXUg6pUKi_qn","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037049.166096_301.1","section":"bucket.instance","name":"tkscyf-27:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.27","timestamp":"2021-02-11T09:50:49.166096762Z","data":{"read_version":{"tag":"_HPEAummOtXzD3iTYGsjL78L","ver":1},"write_version":{"tag":"_HPEAummOtXzD3iTYGsjL78L","ver":2},"status":{"status":"write"}}},{"id":"1_1613037049.171244_302.1","section":"bucket.instance","name":"tkscyf-27:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.27","timestamp":"2021-02-11T09:50:49.171244329Z","data":{"read_version":{"tag":"_HPEAummOtXzD3iTYGsjL78L","ver":1},"write_version":{"tag":"_HPEAummOtXzD3iTYGsjL78L","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037049.219869_303.1","section":"bucket.instance","name":"tkscyf-28:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.28","timestamp":"2021-02-11T09:50:49.219869180Z","data":{"read_version":{"tag":"_-VSdKrlRKUfuwloTfMI6Xuv","ver":1},"write_version":{"tag":"_-VSdKrlRKUfuwloTfMI6Xuv","ver":2},"status":{"status":"write"}}},{"id":"1_1613037049.230173_304.1","section":"bucket.instance","name":"tkscyf-28:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.28","timestamp":"2021-02-11T09:50:49.230173871Z","data":{"read_version":{"tag":"_-VSdKrlRKUfuwloTfMI6Xuv","ver":1},"write_version":{"tag":"_-VSdKrlRKUfuwloTfMI6Xuv","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037049.287108_305.1","section":"bucket.instance","name":"tkscyf-29:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.29","timestamp":"2021-02-11T09:50:49.287108951Z","data":{"read_version":{"tag":"_hyJCgZ3ZqhjZ2B82NLQtbFj","ver":1},"write_version":{"tag":"_hyJCgZ3ZqhjZ2B82NLQtbFj","ver":2},"status":{"status":"write"}}},{"id":"1_1613037049.300847_306.1","section":"bucket.instance","name":"tkscyf-29:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.29","timestamp":"2021-02-11T09:50:49.300847185Z","data":{"read_version":{"tag":"_hyJCgZ3ZqhjZ2B82NLQtbFj","ver":1},"write_version":{"tag":"_hyJCgZ3ZqhjZ2B82NLQtbFj","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037049.364299_307.1","section":"bucket.instance","name":"tkscyf-30:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.30","timestamp":"2021-02-11T09:50:49.364299749Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Es0JugEkT-f3o-riZAdqnrS","ver":1},"status":{"status":"write"}}},{"id":"1_1613037049.370623_308.1","section":"bucket.instance","name":"tkscyf-30:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.30","timestamp":"2021-02-11T09:50:49.370623916Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Es0JugEkT-f3o-riZAdqnrS","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037049.372900_309.1","section":"bucket","name":"tkscyf-30","timestamp":"2021-02-11T09:50:49.372900977Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0ipPDFF-oXK1PAXV9ImeRz8","ver":1},"status":{"status":"write"}}},{"id":"1_1613037049.377959_310.1","section":"bucket","name":"tkscyf-30","timestamp":"2021-02-11T09:50:49.377959725Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0ipPDFF-oXK1PAXV9ImeRz8","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037049.400272_311.1","section":"bucket.instance","name":"tkscyf-31:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.31","timestamp":"2021-02-11T09:50:49.400272843Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_J_JeLJIs-yKAxn3I-jH6tun","ver":1},"status":{"status":"write"}}},{"id":"1_1613037049.406204_312.1","section":"bucket.instance","name":"tkscyf-31:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.31","timestamp":"2021-02-11T09:50:49.406204831Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_J_JeLJIs-yKAxn3I-jH6tun","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037049.407984_313.1","section":"bucket","name":"tkscyf-31","timestamp":"2021-02-11T09:50:49.407984579Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AtSnm2w_MDdPl4ZTuW8BBX1","ver":1},"status":{"status":"write"}}},{"id":"1_1613037049.412626_314.1","section":"bucket","name":"tkscyf-31","timestamp":"2021-02-11T09:50:49.412626629Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AtSnm2w_MDdPl4ZTuW8BBX1","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037049.456098_315.1","section":"bucket.instance","name":"tkscyf-32:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.32","timestamp":"2021-02-11T09:50:49.456098996Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jg7YdCRiL4_wX7QgQ9ZEBCK","ver":1},"status":{"status":"write"}}},{"id":"1_1613037049.461384_316.1","section":"bucket.instance","name":"tkscyf-32:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.32","timestamp":"2021-02-11T09:50:49.461384431Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jg7YdCRiL4_wX7QgQ9ZEBCK","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037049.463339_317.1","section":"bucket","name":"tkscyf-32","timestamp":"2021-02-11T09:50:49.463339008Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eg1axhWTHrzA2NWrcdnReCD","ver":1},"status":{"status":"write"}}},{"id":"1_1613037049.468298_318.1","section":"bucket","name":"tkscyf-32","timestamp":"2021-02-11T09:50:49.468298585Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eg1axhWTHrzA2NWrcdnReCD","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037049.493719_319.1","section":"bucket.instance","name":"tkscyf-30:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.30","timestamp":"2021-02-11T09:50:49.493719070Z","data":{"read_version":{"tag":"_Es0JugEkT-f3o-riZAdqnrS","ver":1},"write_version":{"tag":"_Es0JugEkT-f3o-riZAdqnrS","ver":2},"status":{"status":"write"}}},{"id":"1_1613037049.498976_320.1","section":"bucket.instance","name":"tkscyf-30:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.30","timestamp":"2021-02-11T09:50:49.498976233Z","data":{"read_version":{"tag":"_Es0JugEkT-f3o-riZAdqnrS","ver":1},"write_version":{"tag":"_Es0JugEkT-f3o-riZAdqnrS","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037049.547179_321.1","section":"bucket.instance","name":"tkscyf-31:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.31","timestamp":"2021-02-11T09:50:49.547179938Z","data":{"read_version":{"tag":"_J_JeLJIs-yKAxn3I-jH6tun","ver":1},"write_version":{"tag":"_J_JeLJIs-yKAxn3I-jH6tun","ver":2},"status":{"status":"write"}}},{"id":"1_1613037049.552155_322.1","section":"bucket.instance","name":"tkscyf-31:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.31","timestamp":"2021-02-11T09:50:49.552155032Z","data":{"read_version":{"tag":"_J_JeLJIs-yKAxn3I-jH6tun","ver":1},"write_version":{"tag":"_J_JeLJIs-yKAxn3I-jH6tun","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037049.603090_323.1","section":"bucket.instance","name":"tkscyf-32:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.32","timestamp":"2021-02-11T09:50:49.603090607Z","data":{"read_version":{"tag":"_jg7YdCRiL4_wX7QgQ9ZEBCK","ver":1},"write_version":{"tag":"_jg7YdCRiL4_wX7QgQ9ZEBCK","ver":2},"status":{"status":"write"}}},{"id":"1_1613037049.608976_324.1","section":"bucket.instance","name":"tkscyf-32:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.32","timestamp":"2021-02-11T09:50:49.608976182Z","data":{"read_version":{"tag":"_jg7YdCRiL4_wX7QgQ9ZEBCK","ver":1},"write_version":{"tag":"_jg7YdCRiL4_wX7QgQ9ZEBCK","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037049.680125_325.1","section":"bucket.instance","name":"tkscyf-33:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.33","timestamp":"2021-02-11T09:50:49.680125618Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eZRAP_8ajxLAWs4hu7nqOwz","ver":1},"status":{"status":"write"}}},{"id":"1_1613037049.687075_326.1","section":"bucket.instance","name":"tkscyf-33:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.33","timestamp":"2021-02-11T09:50:49.687075481Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eZRAP_8ajxLAWs4hu7nqOwz","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037049.692386_327.1","section":"bucket","name":"tkscyf-33","timestamp":"2021-02-11T09:50:49.692386405Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-6dR6mEXl2aMVHaPbOLG9sm","ver":1},"status":{"status":"write"}}},{"id":"1_1613037049.699844_328.1","section":"bucket","name":"tkscyf-33","timestamp":"2021-02-11T09:50:49.699844848Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-6dR6mEXl2aMVHaPbOLG9sm","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037049.725396_329.1","section":"bucket.instance","name":"tkscyf-34:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.34","timestamp":"2021-02-11T09:50:49.725396604Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_k7kM2SM39DqWgu3NfoI4bqF","ver":1},"status":{"status":"write"}}},{"id":"1_1613037049.731707_330.1","section":"bucket.instance","name":"tkscyf-34:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.34","timestamp":"2021-02-11T09:50:49.731707348Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_k7kM2SM39DqWgu3NfoI4bqF","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037049.735696_331.1","section":"bucket","name":"tkscyf-34","timestamp":"2021-02-11T09:50:49.735696647Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_L016eB3Fp2Gc1WBb3ihrD9w","ver":1},"status":{"status":"write"}}},{"id":"1_1613037049.740587_332.1","section":"bucket","name":"tkscyf-34","timestamp":"2021-02-11T09:50:49.740587087Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_L016eB3Fp2Gc1WBb3ihrD9w","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037049.780026_333.1","section":"bucket.instance","name":"tkscyf-35:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.35","timestamp":"2021-02-11T09:50:49.780026803Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KosuwqaTc-yYnhDuNK50-jc","ver":1},"status":{"status":"write"}}},{"id":"1_1613037049.785523_334.1","section":"bucket.instance","name":"tkscyf-35:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.35","timestamp":"2021-02-11T09:50:49.785523728Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KosuwqaTc-yYnhDuNK50-jc","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037049.787540_335.1","section":"bucket","name":"tkscyf-35","timestamp":"2021-02-11T09:50:49.787540619Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wBCuU48Ox_uSnLpuXpSWi95","ver":1},"status":{"status":"write"}}},{"id":"1_1613037049.792438_336.1","section":"bucket","name":"tkscyf-35","timestamp":"2021-02-11T09:50:49.792438114Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wBCuU48Ox_uSnLpuXpSWi95","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037049.814263_337.1","section":"bucket.instance","name":"tkscyf-33:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.33","timestamp":"2021-02-11T09:50:49.814263352Z","data":{"read_version":{"tag":"_eZRAP_8ajxLAWs4hu7nqOwz","ver":1},"write_version":{"tag":"_eZRAP_8ajxLAWs4hu7nqOwz","ver":2},"status":{"status":"write"}}},{"id":"1_1613037049.819262_338.1","section":"bucket.instance","name":"tkscyf-33:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.33","timestamp":"2021-02-11T09:50:49.819262498Z","data":{"read_version":{"tag":"_eZRAP_8ajxLAWs4hu7nqOwz","ver":1},"write_version":{"tag":"_eZRAP_8ajxLAWs4hu7nqOwz","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037050.867169_339.1","section":"bucket.instance","name":"tkscyf-34:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.34","timestamp":"2021-02-11T09:50:50.867169930Z","data":{"read_version":{"tag":"_k7kM2SM39DqWgu3NfoI4bqF","ver":1},"write_version":{"tag":"_k7kM2SM39DqWgu3NfoI4bqF","ver":2},"status":{"status":"write"}}},{"id":"1_1613037050.879211_340.1","section":"bucket.instance","name":"tkscyf-34:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.34","timestamp":"2021-02-11T09:50:50.879211406Z","data":{"read_version":{"tag":"_k7kM2SM39DqWgu3NfoI4bqF","ver":1},"write_version":{"tag":"_k7kM2SM39DqWgu3NfoI4bqF","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037051.935225_341.1","section":"bucket.instance","name":"tkscyf-35:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.35","timestamp":"2021-02-11T09:50:51.935225906Z","data":{"read_version":{"tag":"_KosuwqaTc-yYnhDuNK50-jc","ver":1},"write_version":{"tag":"_KosuwqaTc-yYnhDuNK50-jc","ver":2},"status":{"status":"write"}}},{"id":"1_1613037051.946001_342.1","section":"bucket.instance","name":"tkscyf-35:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.35","timestamp":"2021-02-11T09:50:51.946001361Z","data":{"read_version":{"tag":"_KosuwqaTc-yYnhDuNK50-jc","ver":1},"write_version":{"tag":"_KosuwqaTc-yYnhDuNK50-jc","ver":2},"status":{"status":"complete"}}},{"id":"1_1613037056.396292_343.1","section":"bucket.instance","name":"tkscyf-36:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.36","timestamp":"2021-02-11T09:50:56.396292831Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ssZsgWPzFUlbtHH72rboNt2","ver":1},"status":{"status":"write"}}},{"id":"1_1613037056.402315_344.1","section":"bucket.instance","name":"tkscyf-36:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.36","timestamp":"2021-02-11T09:50:56.402315663Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ssZsgWPzFUlbtHH72rboNt2","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037056.404168_345.1","section":"bucket","name":"tkscyf-36","timestamp":"2021-02-11T09:50:56.404168283Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_R9PAwh1OvxD1UjEPuuj8qio","ver":1},"status":{"status":"write"}}},{"id":"1_1613037056.408811_346.1","section":"bucket","name":"tkscyf-36","timestamp":"2021-02-11T09:50:56.408811301Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_R9PAwh1OvxD1UjEPuuj8qio","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037056.437739_347.1","section":"bucket.instance","name":"tkscyf-37:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.37","timestamp":"2021-02-11T09:50:56.437739913Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HWaNKKBo9OCATJepv1PpQ9b","ver":1},"status":{"status":"write"}}},{"id":"1_1613037056.442573_348.1","section":"bucket.instance","name":"tkscyf-37:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.37","timestamp":"2021-02-11T09:50:56.442573229Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HWaNKKBo9OCATJepv1PpQ9b","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037056.444396_349.1","section":"bucket","name":"tkscyf-37","timestamp":"2021-02-11T09:50:56.444396764Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RbeSoBdgaTvi80yVezMNU9a","ver":1},"status":{"status":"write"}}},{"id":"1_1613037056.449127_350.1","section":"bucket","name":"tkscyf-37","timestamp":"2021-02-11T09:50:56.449127942Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RbeSoBdgaTvi80yVezMNU9a","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037056.503850_351.1","section":"bucket.instance","name":"tkscyf-38:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.38","timestamp":"2021-02-11T09:50:56.503850524Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_M7gYPbaihLb1WY3hhZTdAtg","ver":1},"status":{"status":"write"}}},{"id":"1_1613037056.508532_352.1","section":"bucket.instance","name":"tkscyf-38:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.38","timestamp":"2021-02-11T09:50:56.508532698Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_M7gYPbaihLb1WY3hhZTdAtg","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037056.510275_353.1","section":"bucket","name":"tkscyf-38","timestamp":"2021-02-11T09:50:56.510275102Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0RzXHcbVo20pBILfHDUWraY","ver":1},"status":{"status":"write"}}},{"id":"1_1613037056.514482_354.1","section":"bucket","name":"tkscyf-38","timestamp":"2021-02-11T09:50:56.514482141Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0RzXHcbVo20pBILfHDUWraY","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037057.908443_355.1","section":"bucket.instance","name":"tkscyf-39:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.39","timestamp":"2021-02-11T09:50:57.908443576Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_94HMkmvhzlZXLQvruAOVtAJ","ver":1},"status":{"status":"write"}}},{"id":"1_1613037057.914397_356.1","section":"bucket.instance","name":"tkscyf-39:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.39","timestamp":"2021-02-11T09:50:57.914397041Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_94HMkmvhzlZXLQvruAOVtAJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037057.916715_357.1","section":"bucket","name":"tkscyf-39","timestamp":"2021-02-11T09:50:57.916715595Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wsKJ1QOckGXWesrqOCyo1WG","ver":1},"status":{"status":"write"}}},{"id":"1_1613037057.923356_358.1","section":"bucket","name":"tkscyf-39","timestamp":"2021-02-11T09:50:57.923356621Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wsKJ1QOckGXWesrqOCyo1WG","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037057.945478_359.1","section":"bucket.instance","name":"tkscyf-40:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.40","timestamp":"2021-02-11T09:50:57.945478017Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pg3ahh-4pO-_3XyceN-gWsQ","ver":1},"status":{"status":"write"}}},{"id":"1_1613037057.949753_360.1","section":"bucket.instance","name":"tkscyf-40:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.40","timestamp":"2021-02-11T09:50:57.949753049Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pg3ahh-4pO-_3XyceN-gWsQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037057.951444_361.1","section":"bucket","name":"tkscyf-40","timestamp":"2021-02-11T09:50:57.951444977Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ej1h_jFuYjzyoRvb-1C7RAf","ver":1},"status":{"status":"write"}}},{"id":"1_1613037057.955584_362.1","section":"bucket","name":"tkscyf-40","timestamp":"2021-02-11T09:50:57.955584939Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ej1h_jFuYjzyoRvb-1C7RAf","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037057.996014_363.1","section":"bucket.instance","name":"tkscyf-41:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.41","timestamp":"2021-02-11T09:50:57.996014692Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_I9DAcVu7OInplRV1Ta3wruY","ver":1},"status":{"status":"write"}}},{"id":"1_1613037058.000113_364.1","section":"bucket.instance","name":"tkscyf-41:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.41","timestamp":"2021-02-11T09:50:58.000113498Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_I9DAcVu7OInplRV1Ta3wruY","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037058.001757_365.1","section":"bucket","name":"tkscyf-41","timestamp":"2021-02-11T09:50:58.001757095Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_R0j0lRoRH3ZszkT7rysin2m","ver":1},"status":{"status":"write"}}},{"id":"1_1613037058.005761_366.1","section":"bucket","name":"tkscyf-41","timestamp":"2021-02-11T09:50:58.005761416Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_R0j0lRoRH3ZszkT7rysin2m","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036843.308060_40.1","section":"bucket.instance","name":"tkscyf-1:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.1","timestamp":"2021-02-11T09:47:23.308060874Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vJcQgK-3XDRzzGYAO1_aHfA","ver":1},"status":{"status":"write"}}},{"id":"1_1613036843.334677_41.1","section":"bucket.instance","name":"tkscyf-1:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.1","timestamp":"2021-02-11T09:47:23.334677253Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vJcQgK-3XDRzzGYAO1_aHfA","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036843.341061_42.1","section":"bucket","name":"tkscyf-1","timestamp":"2021-02-11T09:47:23.341061810Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_so1GMpfrEZgmi_sTYZSAr8H","ver":1},"status":{"status":"write"}}},{"id":"1_1613036843.345437_43.1","section":"bucket","name":"tkscyf-1","timestamp":"2021-02-11T09:47:23.345437347Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_so1GMpfrEZgmi_sTYZSAr8H","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036843.366232_44.1","section":"bucket.instance","name":"tkscyf-2:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.2","timestamp":"2021-02-11T09:47:23.366232598Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MZ24WdPDpXrrqQPnafXRqiN","ver":1},"status":{"status":"write"}}},{"id":"1_1613036843.371311_45.1","section":"bucket.instance","name":"tkscyf-2:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.2","timestamp":"2021-02-11T09:47:23.371311720Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MZ24WdPDpXrrqQPnafXRqiN","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036843.373088_46.1","section":"bucket","name":"tkscyf-2","timestamp":"2021-02-11T09:47:23.373088017Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5VYuSj_-KrHdRO5ccn8Xe_y","ver":1},"status":{"status":"write"}}},{"id":"1_1613036843.378050_47.1","section":"bucket","name":"tkscyf-2","timestamp":"2021-02-11T09:47:23.378050996Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5VYuSj_-KrHdRO5ccn8Xe_y","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036847.401028_48.1","section":"bucket.instance","name":"tkscyf-3:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.3","timestamp":"2021-02-11T09:47:27.401028055Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ymULWMaifA40PYB81mtiqKO","ver":1},"status":{"status":"write"}}},{"id":"1_1613036847.406630_49.1","section":"bucket.instance","name":"tkscyf-3:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.3","timestamp":"2021-02-11T09:47:27.406630738Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ymULWMaifA40PYB81mtiqKO","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036847.408599_50.1","section":"bucket","name":"tkscyf-3","timestamp":"2021-02-11T09:47:27.408599295Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8F2ONXee867moT-prh9FMvz","ver":1},"status":{"status":"write"}}},{"id":"1_1613036847.414041_51.1","section":"bucket","name":"tkscyf-3","timestamp":"2021-02-11T09:47:27.414041891Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8F2ONXee867moT-prh9FMvz","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036848.515900_52.1","section":"bucket.instance","name":"tkscyf-4:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.4","timestamp":"2021-02-11T09:47:28.515900958Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OC0h4nNzRptiVogF5DDSlbl","ver":1},"status":{"status":"write"}}},{"id":"1_1613036848.521086_53.1","section":"bucket.instance","name":"tkscyf-4:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.4","timestamp":"2021-02-11T09:47:28.521086371Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OC0h4nNzRptiVogF5DDSlbl","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036848.523142_54.1","section":"bucket","name":"tkscyf-4","timestamp":"2021-02-11T09:47:28.523142343Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lzBpaQfMhBTasrckHQnxsrH","ver":1},"status":{"status":"write"}}},{"id":"1_1613036848.528085_55.1","section":"bucket","name":"tkscyf-4","timestamp":"2021-02-11T09:47:28.528085857Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lzBpaQfMhBTasrckHQnxsrH","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036848.545712_56.1","section":"bucket.instance","name":"tkscyf-5:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.5","timestamp":"2021-02-11T09:47:28.545712519Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pizM_4Vxb6ga5d7zQm5cup4","ver":1},"status":{"status":"write"}}},{"id":"1_1613036848.551316_57.1","section":"bucket.instance","name":"tkscyf-5:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.5","timestamp":"2021-02-11T09:47:28.551316982Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pizM_4Vxb6ga5d7zQm5cup4","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036848.553108_58.1","section":"bucket","name":"tkscyf-5","timestamp":"2021-02-11T09:47:28.553108370Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__btqQaGnBiVqesmRij4KpZu","ver":1},"status":{"status":"write"}}},{"id":"1_1613036848.557871_59.1","section":"bucket","name":"tkscyf-5","timestamp":"2021-02-11T09:47:28.557871264Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__btqQaGnBiVqesmRij4KpZu","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036848.598426_60.1","section":"bucket.instance","name":"tkscyf-6:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.6","timestamp":"2021-02-11T09:47:28.598426655Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MXFYvcf4zPQLqC1AyVkpK27","ver":1},"status":{"status":"write"}}},{"id":"1_1613036848.607442_61.1","section":"bucket.instance","name":"tkscyf-6:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.6","timestamp":"2021-02-11T09:47:28.607442762Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MXFYvcf4zPQLqC1AyVkpK27","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036848.609097_62.1","section":"bucket","name":"tkscyf-6","timestamp":"2021-02-11T09:47:28.609097002Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_27D4N2MBNgnn-CUkGMJjydq","ver":1},"status":{"status":"write"}}},{"id":"1_1613036848.613340_63.1","section":"bucket","name":"tkscyf-6","timestamp":"2021-02-11T09:47:28.613340684Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_27D4N2MBNgnn-CUkGMJjydq","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036851.051651_64.1","section":"bucket.instance","name":"tkscyf-7:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.7","timestamp":"2021-02-11T09:47:31.051651058Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qdftsvlxuD5g0Vn1ajzNmyo","ver":1},"status":{"status":"write"}}},{"id":"1_1613036851.056767_65.1","section":"bucket.instance","name":"tkscyf-7:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.7","timestamp":"2021-02-11T09:47:31.056767922Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qdftsvlxuD5g0Vn1ajzNmyo","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036851.058594_66.1","section":"bucket","name":"tkscyf-7","timestamp":"2021-02-11T09:47:31.058594967Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FCpy5wrdCKP58fp9iJu68WQ","ver":1},"status":{"status":"write"}}},{"id":"1_1613036851.063396_67.1","section":"bucket","name":"tkscyf-7","timestamp":"2021-02-11T09:47:31.063396372Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FCpy5wrdCKP58fp9iJu68WQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036851.082324_68.1","section":"bucket.instance","name":"tkscyf-8:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.8","timestamp":"2021-02-11T09:47:31.082324615Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZgPd4GGXu--UcP5ytH49TD4","ver":1},"status":{"status":"write"}}},{"id":"1_1613036851.087320_69.1","section":"bucket.instance","name":"tkscyf-8:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.8","timestamp":"2021-02-11T09:47:31.087320996Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZgPd4GGXu--UcP5ytH49TD4","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036851.089023_70.1","section":"bucket","name":"tkscyf-8","timestamp":"2021-02-11T09:47:31.089023916Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MfurQNSSeZct4L9MZ9BG8Gr","ver":1},"status":{"status":"write"}}},{"id":"1_1613036851.093557_71.1","section":"bucket","name":"tkscyf-8","timestamp":"2021-02-11T09:47:31.093557563Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MfurQNSSeZct4L9MZ9BG8Gr","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036851.133487_72.1","section":"bucket.instance","name":"tkscyf-9:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.9","timestamp":"2021-02-11T09:47:31.133487717Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gym2dhV8yR1PgdklxT3K2xe","ver":1},"status":{"status":"write"}}},{"id":"1_1613036851.138269_73.1","section":"bucket.instance","name":"tkscyf-9:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.9","timestamp":"2021-02-11T09:47:31.138269287Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gym2dhV8yR1PgdklxT3K2xe","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036851.139876_74.1","section":"bucket","name":"tkscyf-9","timestamp":"2021-02-11T09:47:31.139876187Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_p0bbqr2Y572m56xpGPFsiZ-","ver":1},"status":{"status":"write"}}},{"id":"1_1613036851.144668_75.1","section":"bucket","name":"tkscyf-9","timestamp":"2021-02-11T09:47:31.144668547Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_p0bbqr2Y572m56xpGPFsiZ-","ver":1},"status":{"status":"complete"}}},{"id":"1_1613036863.874762_76.1","section":"bucket","name":"tkscyf-7","timestamp":"2021-02-11T09:47:43.874762601Z","data":{"read_version":{"tag":"_FCpy5wrdCKP58fp9iJu68WQ","ver":1},"write_version":{"tag":"_FCpy5wrdCKP58fp9iJu68WQ","ver":2},"status":{"status":"remove"}}},{"id":"1_1613036863.880950_77.1","section":"bucket","name":"tkscyf-7","timestamp":"2021-02-11T09:47:43.880950937Z","data":{"read_version":{"tag":"_FCpy5wrdCKP58fp9iJu68WQ","ver":1},"write_version":{"tag":"_FCpy5wrdCKP58fp9iJu68WQ","ver":2},"status":{"status":"complete"}}},{"id":"1_1613036863.956967_78.1","section":"bucket","name":"tkscyf-8","timestamp":"2021-02-11T09:47:43.956967900Z","data":{"read_version":{"tag":"_MfurQNSSeZct4L9MZ9BG8Gr","ver":1},"write_version":{"tag":"_MfurQNSSeZct4L9MZ9BG8Gr","ver":2},"status":{"status":"remove"}}},{"id":"1_1613036863.963661_79.1","section":"bucket","name":"tkscyf-8","timestamp":"2021-02-11T09:47:43.963661824Z","data":{"read_version":{"tag":"_MfurQNSSeZct4L9MZ9BG8Gr","ver":1},"write_version":{"tag":"_MfurQNSSeZct4L9MZ9BG8Gr","ver":2},"status":{"status":"complete"}}},{"id":"1_1613036864.074054_80.1","section":"bucket","name":"tkscyf-9","timestamp":"2021-02-11T09:47:44.074054305Z","data":{"read_version":{"tag":"_p0bbqr2Y572m56xpGPFsiZ-","ver":1},"write_version":{"tag":"_p0bbqr2Y572m56xpGPFsiZ-","ver":2},"status":{"status":"remove"}}},{"id":"1_1613036864.081584_81.1","section":"bucket","name":"tkscyf-9","timestamp":"2021-02-11T09:47:44.081584235Z","data":{"read_version":{"tag":"_p0bbqr2Y572m56xpGPFsiZ-","ver":1},"write_version":{"tag":"_p0bbqr2Y572m56xpGPFsiZ-","ver":2},"status":{"status":"complete"}}}] 2021-02-11T09:53:15.334 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '47caa906-66a7-4be6-bbb9-9f539accb732', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:15.334 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '47caa906-66a7-4be6-bbb9-9f539accb732', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:15.334 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 47caa906-66a7-4be6-bbb9-9f539accb732 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:15.373 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:15.374 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:15.615 INFO:teuthology.orchestra.run.smithi094.stdout:[{"id":"1_1613037125.654580_104.1","section":"bucket.instance","name":"tkscyf-42:63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2.5092.1","timestamp":"2021-02-11T09:52:05.654580424Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pMEVCZbHJ0choPnFKYtTiWx","ver":1},"status":{"status":"write"}}},{"id":"1_1613037125.660800_110.1","section":"bucket.instance","name":"tkscyf-42:63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2.5092.1","timestamp":"2021-02-11T09:52:05.660800955Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pMEVCZbHJ0choPnFKYtTiWx","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037125.662562_111.1","section":"bucket","name":"tkscyf-42","timestamp":"2021-02-11T09:52:05.662562378Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CXMCEkoDP0lnk6q0wMdtx4z","ver":1},"status":{"status":"write"}}},{"id":"1_1613037125.667464_112.1","section":"bucket","name":"tkscyf-42","timestamp":"2021-02-11T09:52:05.667464270Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CXMCEkoDP0lnk6q0wMdtx4z","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037125.714462_113.1","section":"bucket.instance","name":"tkscyf-43:63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2.5092.2","timestamp":"2021-02-11T09:52:05.714462722Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_q8i5VsnPr3gATS0158v32a5","ver":1},"status":{"status":"write"}}},{"id":"1_1613037125.721233_114.1","section":"bucket.instance","name":"tkscyf-43:63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2.5092.2","timestamp":"2021-02-11T09:52:05.721233986Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_q8i5VsnPr3gATS0158v32a5","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037125.722914_115.1","section":"bucket","name":"tkscyf-43","timestamp":"2021-02-11T09:52:05.722914863Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JyQ-Yw2jq6PtQdN0gl0FXRR","ver":1},"status":{"status":"write"}}},{"id":"1_1613037125.727813_116.1","section":"bucket","name":"tkscyf-43","timestamp":"2021-02-11T09:52:05.727813911Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JyQ-Yw2jq6PtQdN0gl0FXRR","ver":1},"status":{"status":"complete"}}}] 2021-02-11T09:53:15.616 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '47caa906-66a7-4be6-bbb9-9f539accb732', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:15.616 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '47caa906-66a7-4be6-bbb9-9f539accb732', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:15.616 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 47caa906-66a7-4be6-bbb9-9f539accb732 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:15.656 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:15.656 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:15.913 INFO:teuthology.orchestra.run.smithi098.stdout:[{"id":"1_1613037125.654580_104.1","section":"bucket.instance","name":"tkscyf-42:63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2.5092.1","timestamp":"2021-02-11T09:52:05.654580424Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pMEVCZbHJ0choPnFKYtTiWx","ver":1},"status":{"status":"write"}}},{"id":"1_1613037125.660800_110.1","section":"bucket.instance","name":"tkscyf-42:63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2.5092.1","timestamp":"2021-02-11T09:52:05.660800955Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pMEVCZbHJ0choPnFKYtTiWx","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037125.662562_111.1","section":"bucket","name":"tkscyf-42","timestamp":"2021-02-11T09:52:05.662562378Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CXMCEkoDP0lnk6q0wMdtx4z","ver":1},"status":{"status":"write"}}},{"id":"1_1613037125.667464_112.1","section":"bucket","name":"tkscyf-42","timestamp":"2021-02-11T09:52:05.667464270Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CXMCEkoDP0lnk6q0wMdtx4z","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037125.714462_113.1","section":"bucket.instance","name":"tkscyf-43:63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2.5092.2","timestamp":"2021-02-11T09:52:05.714462722Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_q8i5VsnPr3gATS0158v32a5","ver":1},"status":{"status":"write"}}},{"id":"1_1613037125.721233_114.1","section":"bucket.instance","name":"tkscyf-43:63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2.5092.2","timestamp":"2021-02-11T09:52:05.721233986Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_q8i5VsnPr3gATS0158v32a5","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037125.722914_115.1","section":"bucket","name":"tkscyf-43","timestamp":"2021-02-11T09:52:05.722914863Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JyQ-Yw2jq6PtQdN0gl0FXRR","ver":1},"status":{"status":"write"}}},{"id":"1_1613037125.727813_116.1","section":"bucket","name":"tkscyf-43","timestamp":"2021-02-11T09:52:05.727813911Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JyQ-Yw2jq6PtQdN0gl0FXRR","ver":1},"status":{"status":"complete"}}}] 2021-02-11T09:53:15.915 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '47caa906-66a7-4be6-bbb9-9f539accb732', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:15.916 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '47caa906-66a7-4be6-bbb9-9f539accb732', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:15.916 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 47caa906-66a7-4be6-bbb9-9f539accb732 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:15.956 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:15.957 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:16.195 INFO:teuthology.orchestra.run.smithi094.stdout:[{"id":"1_1613037125.654580_104.1","section":"bucket.instance","name":"tkscyf-42:63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2.5092.1","timestamp":"2021-02-11T09:52:05.654580424Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pMEVCZbHJ0choPnFKYtTiWx","ver":1},"status":{"status":"write"}}},{"id":"1_1613037125.660800_110.1","section":"bucket.instance","name":"tkscyf-42:63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2.5092.1","timestamp":"2021-02-11T09:52:05.660800955Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pMEVCZbHJ0choPnFKYtTiWx","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037125.662562_111.1","section":"bucket","name":"tkscyf-42","timestamp":"2021-02-11T09:52:05.662562378Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CXMCEkoDP0lnk6q0wMdtx4z","ver":1},"status":{"status":"write"}}},{"id":"1_1613037125.667464_112.1","section":"bucket","name":"tkscyf-42","timestamp":"2021-02-11T09:52:05.667464270Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CXMCEkoDP0lnk6q0wMdtx4z","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037125.714462_113.1","section":"bucket.instance","name":"tkscyf-43:63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2.5092.2","timestamp":"2021-02-11T09:52:05.714462722Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_q8i5VsnPr3gATS0158v32a5","ver":1},"status":{"status":"write"}}},{"id":"1_1613037125.721233_114.1","section":"bucket.instance","name":"tkscyf-43:63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2.5092.2","timestamp":"2021-02-11T09:52:05.721233986Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_q8i5VsnPr3gATS0158v32a5","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037125.722914_115.1","section":"bucket","name":"tkscyf-43","timestamp":"2021-02-11T09:52:05.722914863Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JyQ-Yw2jq6PtQdN0gl0FXRR","ver":1},"status":{"status":"write"}}},{"id":"1_1613037125.727813_116.1","section":"bucket","name":"tkscyf-43","timestamp":"2021-02-11T09:52:05.727813911Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JyQ-Yw2jq6PtQdN0gl0FXRR","ver":1},"status":{"status":"complete"}}}] 2021-02-11T09:53:16.196 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '47caa906-66a7-4be6-bbb9-9f539accb732', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:16.196 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '47caa906-66a7-4be6-bbb9-9f539accb732', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:16.196 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 47caa906-66a7-4be6-bbb9-9f539accb732 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:16.238 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:16.238 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:16.485 INFO:teuthology.orchestra.run.smithi098.stdout:[{"id":"1_1613037125.654580_104.1","section":"bucket.instance","name":"tkscyf-42:63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2.5092.1","timestamp":"2021-02-11T09:52:05.654580424Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pMEVCZbHJ0choPnFKYtTiWx","ver":1},"status":{"status":"write"}}},{"id":"1_1613037125.660800_110.1","section":"bucket.instance","name":"tkscyf-42:63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2.5092.1","timestamp":"2021-02-11T09:52:05.660800955Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pMEVCZbHJ0choPnFKYtTiWx","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037125.662562_111.1","section":"bucket","name":"tkscyf-42","timestamp":"2021-02-11T09:52:05.662562378Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CXMCEkoDP0lnk6q0wMdtx4z","ver":1},"status":{"status":"write"}}},{"id":"1_1613037125.667464_112.1","section":"bucket","name":"tkscyf-42","timestamp":"2021-02-11T09:52:05.667464270Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CXMCEkoDP0lnk6q0wMdtx4z","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037125.714462_113.1","section":"bucket.instance","name":"tkscyf-43:63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2.5092.2","timestamp":"2021-02-11T09:52:05.714462722Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_q8i5VsnPr3gATS0158v32a5","ver":1},"status":{"status":"write"}}},{"id":"1_1613037125.721233_114.1","section":"bucket.instance","name":"tkscyf-43:63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2.5092.2","timestamp":"2021-02-11T09:52:05.721233986Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_q8i5VsnPr3gATS0158v32a5","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037125.722914_115.1","section":"bucket","name":"tkscyf-43","timestamp":"2021-02-11T09:52:05.722914863Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JyQ-Yw2jq6PtQdN0gl0FXRR","ver":1},"status":{"status":"write"}}},{"id":"1_1613037125.727813_116.1","section":"bucket","name":"tkscyf-43","timestamp":"2021-02-11T09:52:05.727813911Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JyQ-Yw2jq6PtQdN0gl0FXRR","ver":1},"status":{"status":"complete"}}}] 2021-02-11T09:53:16.486 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '0d37c290-12ce-40b6-8dd3-c3f1d036f1a7', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:16.486 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '0d37c290-12ce-40b6-8dd3-c3f1d036f1a7', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:16.486 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:16.525 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:16.525 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:16.759 INFO:teuthology.orchestra.run.smithi094.stdout:[{"id":"1_1613037186.828232_242.1","section":"bucket.instance","name":"tkscyf-44:dd308248-8ee8-49be-b928-07c3de7d7aa2.5145.1","timestamp":"2021-02-11T09:53:06.828232105Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nruhKoeLhbSGk4uAXBSCaav","ver":1},"status":{"status":"write"}}},{"id":"1_1613037186.835102_243.1","section":"bucket.instance","name":"tkscyf-44:dd308248-8ee8-49be-b928-07c3de7d7aa2.5145.1","timestamp":"2021-02-11T09:53:06.835102683Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nruhKoeLhbSGk4uAXBSCaav","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037186.837054_244.1","section":"bucket","name":"tkscyf-44","timestamp":"2021-02-11T09:53:06.837054532Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YyxMWkM1jw6elu2oR0v1tJx","ver":1},"status":{"status":"write"}}},{"id":"1_1613037186.841844_245.1","section":"bucket","name":"tkscyf-44","timestamp":"2021-02-11T09:53:06.841844052Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YyxMWkM1jw6elu2oR0v1tJx","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037186.878472_246.1","section":"bucket.instance","name":"tkscyf-45:dd308248-8ee8-49be-b928-07c3de7d7aa2.5145.2","timestamp":"2021-02-11T09:53:06.878472707Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1djVO35zJOQpasqHe297gbo","ver":1},"status":{"status":"write"}}},{"id":"1_1613037186.884079_247.1","section":"bucket.instance","name":"tkscyf-45:dd308248-8ee8-49be-b928-07c3de7d7aa2.5145.2","timestamp":"2021-02-11T09:53:06.884079468Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1djVO35zJOQpasqHe297gbo","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037186.886021_248.1","section":"bucket","name":"tkscyf-45","timestamp":"2021-02-11T09:53:06.886021720Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zjzyO0akX49OTxHaYozxH45","ver":1},"status":{"status":"write"}}},{"id":"1_1613037186.890832_249.1","section":"bucket","name":"tkscyf-45","timestamp":"2021-02-11T09:53:06.890832862Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zjzyO0akX49OTxHaYozxH45","ver":1},"status":{"status":"complete"}}}] 2021-02-11T09:53:16.760 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '0d37c290-12ce-40b6-8dd3-c3f1d036f1a7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:16.760 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '0d37c290-12ce-40b6-8dd3-c3f1d036f1a7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:16.760 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:16.799 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:16.799 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:17.048 INFO:teuthology.orchestra.run.smithi098.stdout:[{"id":"1_1613037186.828232_242.1","section":"bucket.instance","name":"tkscyf-44:dd308248-8ee8-49be-b928-07c3de7d7aa2.5145.1","timestamp":"2021-02-11T09:53:06.828232105Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nruhKoeLhbSGk4uAXBSCaav","ver":1},"status":{"status":"write"}}},{"id":"1_1613037186.835102_243.1","section":"bucket.instance","name":"tkscyf-44:dd308248-8ee8-49be-b928-07c3de7d7aa2.5145.1","timestamp":"2021-02-11T09:53:06.835102683Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nruhKoeLhbSGk4uAXBSCaav","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037186.837054_244.1","section":"bucket","name":"tkscyf-44","timestamp":"2021-02-11T09:53:06.837054532Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YyxMWkM1jw6elu2oR0v1tJx","ver":1},"status":{"status":"write"}}},{"id":"1_1613037186.841844_245.1","section":"bucket","name":"tkscyf-44","timestamp":"2021-02-11T09:53:06.841844052Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YyxMWkM1jw6elu2oR0v1tJx","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037186.878472_246.1","section":"bucket.instance","name":"tkscyf-45:dd308248-8ee8-49be-b928-07c3de7d7aa2.5145.2","timestamp":"2021-02-11T09:53:06.878472707Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1djVO35zJOQpasqHe297gbo","ver":1},"status":{"status":"write"}}},{"id":"1_1613037186.884079_247.1","section":"bucket.instance","name":"tkscyf-45:dd308248-8ee8-49be-b928-07c3de7d7aa2.5145.2","timestamp":"2021-02-11T09:53:06.884079468Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1djVO35zJOQpasqHe297gbo","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037186.886021_248.1","section":"bucket","name":"tkscyf-45","timestamp":"2021-02-11T09:53:06.886021720Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zjzyO0akX49OTxHaYozxH45","ver":1},"status":{"status":"write"}}},{"id":"1_1613037186.890832_249.1","section":"bucket","name":"tkscyf-45","timestamp":"2021-02-11T09:53:06.890832862Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zjzyO0akX49OTxHaYozxH45","ver":1},"status":{"status":"complete"}}}] 2021-02-11T09:53:17.049 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '0d37c290-12ce-40b6-8dd3-c3f1d036f1a7', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:17.049 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '0d37c290-12ce-40b6-8dd3-c3f1d036f1a7', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:17.049 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:17.086 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:17.086 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:17.321 INFO:teuthology.orchestra.run.smithi094.stdout:[{"id":"1_1613037186.828232_242.1","section":"bucket.instance","name":"tkscyf-44:dd308248-8ee8-49be-b928-07c3de7d7aa2.5145.1","timestamp":"2021-02-11T09:53:06.828232105Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nruhKoeLhbSGk4uAXBSCaav","ver":1},"status":{"status":"write"}}},{"id":"1_1613037186.835102_243.1","section":"bucket.instance","name":"tkscyf-44:dd308248-8ee8-49be-b928-07c3de7d7aa2.5145.1","timestamp":"2021-02-11T09:53:06.835102683Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nruhKoeLhbSGk4uAXBSCaav","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037186.837054_244.1","section":"bucket","name":"tkscyf-44","timestamp":"2021-02-11T09:53:06.837054532Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YyxMWkM1jw6elu2oR0v1tJx","ver":1},"status":{"status":"write"}}},{"id":"1_1613037186.841844_245.1","section":"bucket","name":"tkscyf-44","timestamp":"2021-02-11T09:53:06.841844052Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YyxMWkM1jw6elu2oR0v1tJx","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037186.878472_246.1","section":"bucket.instance","name":"tkscyf-45:dd308248-8ee8-49be-b928-07c3de7d7aa2.5145.2","timestamp":"2021-02-11T09:53:06.878472707Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1djVO35zJOQpasqHe297gbo","ver":1},"status":{"status":"write"}}},{"id":"1_1613037186.884079_247.1","section":"bucket.instance","name":"tkscyf-45:dd308248-8ee8-49be-b928-07c3de7d7aa2.5145.2","timestamp":"2021-02-11T09:53:06.884079468Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1djVO35zJOQpasqHe297gbo","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037186.886021_248.1","section":"bucket","name":"tkscyf-45","timestamp":"2021-02-11T09:53:06.886021720Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zjzyO0akX49OTxHaYozxH45","ver":1},"status":{"status":"write"}}},{"id":"1_1613037186.890832_249.1","section":"bucket","name":"tkscyf-45","timestamp":"2021-02-11T09:53:06.890832862Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zjzyO0akX49OTxHaYozxH45","ver":1},"status":{"status":"complete"}}}] 2021-02-11T09:53:17.321 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '0d37c290-12ce-40b6-8dd3-c3f1d036f1a7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:17.321 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '0d37c290-12ce-40b6-8dd3-c3f1d036f1a7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:17.321 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:17.355 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:17.355 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:17.592 INFO:teuthology.orchestra.run.smithi098.stdout:[{"id":"1_1613037186.828232_242.1","section":"bucket.instance","name":"tkscyf-44:dd308248-8ee8-49be-b928-07c3de7d7aa2.5145.1","timestamp":"2021-02-11T09:53:06.828232105Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nruhKoeLhbSGk4uAXBSCaav","ver":1},"status":{"status":"write"}}},{"id":"1_1613037186.835102_243.1","section":"bucket.instance","name":"tkscyf-44:dd308248-8ee8-49be-b928-07c3de7d7aa2.5145.1","timestamp":"2021-02-11T09:53:06.835102683Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nruhKoeLhbSGk4uAXBSCaav","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037186.837054_244.1","section":"bucket","name":"tkscyf-44","timestamp":"2021-02-11T09:53:06.837054532Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YyxMWkM1jw6elu2oR0v1tJx","ver":1},"status":{"status":"write"}}},{"id":"1_1613037186.841844_245.1","section":"bucket","name":"tkscyf-44","timestamp":"2021-02-11T09:53:06.841844052Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YyxMWkM1jw6elu2oR0v1tJx","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037186.878472_246.1","section":"bucket.instance","name":"tkscyf-45:dd308248-8ee8-49be-b928-07c3de7d7aa2.5145.2","timestamp":"2021-02-11T09:53:06.878472707Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1djVO35zJOQpasqHe297gbo","ver":1},"status":{"status":"write"}}},{"id":"1_1613037186.884079_247.1","section":"bucket.instance","name":"tkscyf-45:dd308248-8ee8-49be-b928-07c3de7d7aa2.5145.2","timestamp":"2021-02-11T09:53:06.884079468Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1djVO35zJOQpasqHe297gbo","ver":1},"status":{"status":"complete"}}},{"id":"1_1613037186.886021_248.1","section":"bucket","name":"tkscyf-45","timestamp":"2021-02-11T09:53:06.886021720Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zjzyO0akX49OTxHaYozxH45","ver":1},"status":{"status":"write"}}},{"id":"1_1613037186.890832_249.1","section":"bucket","name":"tkscyf-45","timestamp":"2021-02-11T09:53:06.890832862Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zjzyO0akX49OTxHaYozxH45","ver":1},"status":{"status":"complete"}}}] 2021-02-11T09:53:17.593 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:53:17.593 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:53:17.593 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T09:53:17.628 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:17.628 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:17.901 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:53:17.901 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:53:17.901 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T09:53:17.936 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:17.937 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:18.351 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:53:18.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:53:18.352 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T09:53:18.387 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:18.388 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:18.627 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:53:18.627 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:53:18.628 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T09:53:18.663 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:18.663 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:19.061 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:19.062 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:19.062 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:19.101 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:19.101 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:19.324 INFO:teuthology.orchestra.run.smithi094.stdout:[] 2021-02-11T09:53:19.324 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:19.324 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:19.324 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:19.362 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:19.362 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:19.609 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2021-02-11T09:53:19.609 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:19.609 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:19.610 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:19.649 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:19.649 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:19.879 INFO:teuthology.orchestra.run.smithi094.stdout:[] 2021-02-11T09:53:19.879 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:19.879 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:19.880 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:19.918 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:19.919 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:20.180 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2021-02-11T09:53:20.180 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '47caa906-66a7-4be6-bbb9-9f539accb732', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:20.180 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '47caa906-66a7-4be6-bbb9-9f539accb732', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:20.180 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 47caa906-66a7-4be6-bbb9-9f539accb732 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:20.218 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:20.218 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:20.452 INFO:teuthology.orchestra.run.smithi094.stdout:[] 2021-02-11T09:53:20.452 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '47caa906-66a7-4be6-bbb9-9f539accb732', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:20.452 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '47caa906-66a7-4be6-bbb9-9f539accb732', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:20.453 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 47caa906-66a7-4be6-bbb9-9f539accb732 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:20.492 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:20.493 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:20.740 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2021-02-11T09:53:20.740 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '47caa906-66a7-4be6-bbb9-9f539accb732', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:20.740 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '47caa906-66a7-4be6-bbb9-9f539accb732', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:20.741 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 47caa906-66a7-4be6-bbb9-9f539accb732 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:20.778 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:20.778 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:21.022 INFO:teuthology.orchestra.run.smithi094.stdout:[] 2021-02-11T09:53:21.023 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '47caa906-66a7-4be6-bbb9-9f539accb732', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:21.023 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '47caa906-66a7-4be6-bbb9-9f539accb732', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:21.023 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 47caa906-66a7-4be6-bbb9-9f539accb732 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:21.064 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:21.064 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:21.315 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2021-02-11T09:53:21.316 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '0d37c290-12ce-40b6-8dd3-c3f1d036f1a7', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:21.316 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '0d37c290-12ce-40b6-8dd3-c3f1d036f1a7', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:21.316 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:21.353 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:21.354 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:21.579 INFO:teuthology.orchestra.run.smithi094.stdout:[] 2021-02-11T09:53:21.579 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '0d37c290-12ce-40b6-8dd3-c3f1d036f1a7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:21.580 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '0d37c290-12ce-40b6-8dd3-c3f1d036f1a7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:21.580 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:21.618 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:21.619 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:21.866 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2021-02-11T09:53:21.867 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '0d37c290-12ce-40b6-8dd3-c3f1d036f1a7', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:21.867 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '0d37c290-12ce-40b6-8dd3-c3f1d036f1a7', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:21.867 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:21.907 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:21.907 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:22.140 INFO:teuthology.orchestra.run.smithi094.stdout:[] 2021-02-11T09:53:22.140 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '0d37c290-12ce-40b6-8dd3-c3f1d036f1a7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:22.140 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '0d37c290-12ce-40b6-8dd3-c3f1d036f1a7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:22.140 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:22.180 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:22.180 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:22.414 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2021-02-11T09:53:22.415 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_multi_period_incremental_sync ... ok 2021-02-11T09:53:22.415 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=tkscyf-46 2021-02-11T09:53:22.452 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=tkscyf-47 2021-02-11T09:53:22.522 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=tkscyf-48 2021-02-11T09:53:22.627 INFO:tasks.rgw.c1.client.1.smithi094.stdout:2021-02-11T09:53:22.625+0000 7f8fa97fa700 -1 data sync zone:dd308248 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T09:53:22.630 INFO:tasks.rgw.c2.client.1.smithi098.stdout:2021-02-11T09:53:22.624+0000 7fa813ff7700 -1 data sync zone:dd308248 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T09:53:22.633 INFO:tasks.rgw.c2.client.0.smithi098.stdout:2021-02-11T09:53:22.628+0000 7f2bb37e6700 -1 data sync zone:dd308248 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T09:53:22.722 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:22.723 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:22.723 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:22.762 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:22.762 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:22.913 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 2021-02-11T09:53:22.943 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613037202.556753_281.1","last_update":"2021-02-11T09:53:22.556753Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T09:53:22.943 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T09:53:22.944 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:22.944 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:22.944 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:22.986 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:22.986 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:23.222 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037202.556753_281.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:53:22.556753Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:53:23.223 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T09:53:23.223 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T09:53:23.223 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:23.223 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:23.223 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:23.268 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:23.268 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:23.475 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037202.556753_281.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:53:22.556753Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:53:23.475 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T09:53:23.475 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T09:53:23.475 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:23.476 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:23.476 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:23.516 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:23.516 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:23.776 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037202.556753_281.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:53:22.556753Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:53:23.777 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T09:53:23.777 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:23.777 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:23.777 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:23.817 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:23.817 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:24.043 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613036964.844780_237.1","last_update":"2021-02-11T09:49:24.844780Z"},{"marker":"1_1613037058.105459_155.1","last_update":"2021-02-11T09:50:58.105459Z"},{"marker":"1_1613036942.835890_67.3","last_update":"2021-02-11T09:49:02.835890Z"},{"marker":"1_1613037202.869834_168.1","last_update":"2021-02-11T09:53:22.869834Z"}] 2021-02-11T09:53:24.043 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-02-11T09:53:24.044 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:24.044 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:24.044 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:24.088 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:24.088 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:24.335 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4703921989677143155},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613036964.844780_237.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:49:24.844780Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037058.105459_155.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:50:58.105459Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613036942.835890_67.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:49:02.835890Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037202.869834_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:53:22.869834Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T09:53:24.336 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-02-11T09:53:24.336 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:24.336 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:24.336 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:24.371 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:24.371 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:24.604 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613036964.844780_237.1","last_update":"2021-02-11T09:49:24.844780Z"},{"marker":"1_1613037058.105459_155.1","last_update":"2021-02-11T09:50:58.105459Z"},{"marker":"1_1613036942.835890_67.3","last_update":"2021-02-11T09:49:02.835890Z"},{"marker":"1_1613037202.869834_168.1","last_update":"2021-02-11T09:53:22.869834Z"}] 2021-02-11T09:53:24.604 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-02-11T09:53:24.604 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:24.604 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:24.604 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:24.686 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:24.686 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:24.911 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":13312279424886386302},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613036964.844780_237.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:49:24.844780Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037058.105459_155.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:50:58.105459Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613036942.835890_67.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:49:02.835890Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037202.869834_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:53:22.869834Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T09:53:24.911 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-02-11T09:53:24.912 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:24.912 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:24.912 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:24.991 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:24.991 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:25.210 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613036964.844780_237.1","last_update":"2021-02-11T09:49:24.844780Z"},{"marker":"1_1613037058.105459_155.1","last_update":"2021-02-11T09:50:58.105459Z"},{"marker":"1_1613036942.835890_67.3","last_update":"2021-02-11T09:49:02.835890Z"},{"marker":"1_1613037202.869834_168.1","last_update":"2021-02-11T09:53:22.869834Z"}] 2021-02-11T09:53:25.211 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-02-11T09:53:25.211 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:25.211 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:25.211 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:25.253 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:25.253 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:25.473 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2896161496309628191},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613036964.844780_237.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:49:24.844780Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037058.105459_155.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:50:58.105459Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613036942.835890_67.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:49:02.835890Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037202.869834_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:53:22.869834Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T09:53:25.473 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-02-11T09:53:25.473 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:25.474 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:25.474 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:25.552 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:25.552 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:25.795 INFO:teuthology.orchestra.run.smithi098.stdout:[{"marker":"1_1613036964.822510_109.1","last_update":"2021-02-11T09:49:24.822510Z"},{"marker":"1_1613037058.036305_154.1","last_update":"2021-02-11T09:50:58.036305Z"},{"marker":"1_1613036942.814064_78.3","last_update":"2021-02-11T09:49:02.814064Z"},{"marker":"1_1613037202.851745_175.1","last_update":"2021-02-11T09:53:22.851745Z"}] 2021-02-11T09:53:25.796 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-02-11T09:53:25.796 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:25.796 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:25.796 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:25.839 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:25.839 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:26.076 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":15226599365335882107},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613036964.822510_109.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:49:24.822510Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037058.036305_154.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:50:58.036305Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613036942.814064_78.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:49:02.814064Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037202.851745_175.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:53:22.851745Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T09:53:26.076 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-02-11T09:53:26.076 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:26.076 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:26.077 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:26.115 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:26.115 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:26.368 INFO:teuthology.orchestra.run.smithi098.stdout:[{"marker":"1_1613036964.822510_109.1","last_update":"2021-02-11T09:49:24.822510Z"},{"marker":"1_1613037058.036305_154.1","last_update":"2021-02-11T09:50:58.036305Z"},{"marker":"1_1613036942.814064_78.3","last_update":"2021-02-11T09:49:02.814064Z"},{"marker":"1_1613037202.851745_175.1","last_update":"2021-02-11T09:53:22.851745Z"}] 2021-02-11T09:53:26.368 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-02-11T09:53:26.368 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:26.368 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:26.369 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:26.407 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:26.407 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:26.638 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":10999913493642558030},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613036964.822510_109.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:49:24.822510Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037058.036305_154.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:50:58.036305Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613036942.814064_78.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:49:02.814064Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037202.851745_175.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:53:22.851745Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T09:53:26.638 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-02-11T09:53:26.639 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:26.639 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:26.639 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:26.720 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:26.720 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:26.938 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613036964.875382_129.1","last_update":"2021-02-11T09:49:24.875382Z"},{"marker":"1_1613037057.864761_112.1","last_update":"2021-02-11T09:50:57.864761Z"},{"marker":"1_1613036942.869312_60.3","last_update":"2021-02-11T09:49:02.869312Z"},{"marker":"1_1613037202.861235_109.1","last_update":"2021-02-11T09:53:22.861235Z"}] 2021-02-11T09:53:26.939 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-02-11T09:53:26.939 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:26.939 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:26.939 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:27.022 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:27.022 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:27.242 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2753764877348348355},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613036964.875382_129.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:49:24.875382Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037057.864761_112.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:50:57.864761Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613036942.869312_60.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:49:02.869312Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037052.887653_88.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:50:52.887653Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T09:53:27.243 WARNING:rgw_multi.tests:data of zone test-zone1 behind zone test-zone3: shard=3 master=1_1613037202.861235_109.1 target=1_1613037052.887653_88.3 2021-02-11T09:53:32.244 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:32.244 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:32.244 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:32.286 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:32.286 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:32.519 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2753764877348348355},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613036964.875382_129.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:49:24.875382Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037057.864761_112.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:50:57.864761Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613036942.869312_60.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:49:02.869312Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037202.861235_109.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:53:22.861235Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T09:53:32.520 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-02-11T09:53:32.520 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:32.520 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:32.520 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:32.601 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:32.601 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:32.834 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613036964.875382_129.1","last_update":"2021-02-11T09:49:24.875382Z"},{"marker":"1_1613037057.864761_112.1","last_update":"2021-02-11T09:50:57.864761Z"},{"marker":"1_1613036942.869312_60.3","last_update":"2021-02-11T09:49:02.869312Z"},{"marker":"1_1613037202.861235_109.1","last_update":"2021-02-11T09:53:22.861235Z"}] 2021-02-11T09:53:32.834 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-02-11T09:53:32.834 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:32.834 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:32.835 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:32.879 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:32.879 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:33.133 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":7143279782539285748},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613036964.875382_129.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:49:24.875382Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037057.864761_112.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:50:57.864761Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613036942.869312_60.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:49:02.869312Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037202.861235_109.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:53:22.861235Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T09:53:33.134 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-02-11T09:53:33.134 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:33.134 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:33.134 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:33.170 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:33.170 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:33.394 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613036964.844780_237.1","last_update":"2021-02-11T09:49:24.844780Z"},{"marker":"1_1613037058.105459_155.1","last_update":"2021-02-11T09:50:58.105459Z"},{"marker":"1_1613036942.835890_67.3","last_update":"2021-02-11T09:49:02.835890Z"},{"marker":"1_1613037207.348502_176.1","last_update":"2021-02-11T09:53:27.348502Z"}] 2021-02-11T09:53:33.394 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:53:33.394 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:53:33.395 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T09:53:33.474 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:33.474 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:33.721 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:33.721 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:33.722 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:33.759 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:33.760 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:34.006 INFO:teuthology.orchestra.run.smithi098.stdout:[{"marker":"1_1613036964.822510_109.1","last_update":"2021-02-11T09:49:24.822510Z"},{"marker":"1_1613037058.036305_154.1","last_update":"2021-02-11T09:50:58.036305Z"},{"marker":"1_1613036942.814064_78.3","last_update":"2021-02-11T09:49:02.814064Z"},{"marker":"1_1613037207.242275_182.1","last_update":"2021-02-11T09:53:27.242275Z"}] 2021-02-11T09:53:34.006 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:53:34.006 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:53:34.007 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T09:53:34.082 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:34.082 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:34.343 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:34.343 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:53:34.343 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:53:34.379 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:34.380 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:34.614 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613036964.844780_237.1","last_update":"2021-02-11T09:49:24.844780Z"},{"marker":"1_1613037058.105459_155.1","last_update":"2021-02-11T09:50:58.105459Z"},{"marker":"1_1613036942.835890_67.3","last_update":"2021-02-11T09:49:02.835890Z"},{"marker":"1_1613037207.348502_176.1","last_update":"2021-02-11T09:53:27.348502Z"}] 2021-02-11T09:53:34.615 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:53:34.615 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:53:34.615 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T09:53:34.693 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:34.693 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:34.934 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_datalog_autotrim ... ok 2021-02-11T09:53:34.934 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:53:34.935 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:53:34.935 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --sync-from-all=false --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T09:53:34.975 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:34.975 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:35.004 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:53:35.000+0000 7f019cfc1080 0 failed reading obj info from .rgw.root:zone_info.92d097d8-38ae-4839-9777-3b30db2b2f60: (2) No such file or directory 2021-02-11T09:53:35.004 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:53:35.000+0000 7f019cfc1080 0 WARNING: could not read zone params for zone id=92d097d8-38ae-4839-9777-3b30db2b2f60 name=test-zone3 2021-02-11T09:53:35.005 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:53:35.000+0000 7f019cfc1080 0 failed reading obj info from .rgw.root:zone_info.dd308248-8ee8-49be-b928-07c3de7d7aa2: (2) No such file or directory 2021-02-11T09:53:35.005 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:53:35.000+0000 7f019cfc1080 0 WARNING: could not read zone params for zone id=dd308248-8ee8-49be-b928-07c3de7d7aa2 name=test-zone1 2021-02-11T09:53:35.019 INFO:teuthology.orchestra.run.smithi098.stdout:{"id":"63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"7B4L28ZSM4TSLFS6","secret_key":"eZwNt56BcNJm7h6ZlwjNkhL3ytjGBlGP"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"7e807acb-79b4-4a09-ba75-06bbedad4839","notif_pool":"test-zone2.rgw.log:notif"} 2021-02-11T09:53:35.019 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:53:35.019 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:53:35.020 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T09:53:35.103 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:53:35.103 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:53:35.367 INFO:teuthology.orchestra.run.smithi098.stderr:Sending period to new master zone dd308248-8ee8-49be-b928-07c3de7d7aa2 2021-02-11T09:53:35.457 INFO:teuthology.orchestra.run.smithi098.stdout:{"id":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","epoch":2,"predecessor_uuid":"47caa906-66a7-4be6-bbb9-9f539accb732","sync_status":["1_1613037125.727813_116.1","","",""],"period_map":{"id":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi094.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"dd308248-8ee8-49be-b928-07c3de7d7aa2","zones":[{"id":"45265506-b5d1-4710-9a9e-3e67bee0e451","name":"test-zone4","endpoints":["http://smithi098.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2","name":"test-zone2","endpoints":["http://smithi098.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"false","sync_from":[],"redirect_zone":""},{"id":"92d097d8-38ae-4839-9777-3b30db2b2f60","name":"test-zone3","endpoints":["http://smithi094.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"dd308248-8ee8-49be-b928-07c3de7d7aa2","name":"test-zone1","endpoints":["http://smithi094.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"7e807acb-79b4-4a09-ba75-06bbedad4839","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"45265506-b5d1-4710-9a9e-3e67bee0e451","val":240795374},{"key":"63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2","val":1552258706},{"key":"92d097d8-38ae-4839-9777-3b30db2b2f60","val":2008974102},{"key":"dd308248-8ee8-49be-b928-07c3de7d7aa2","val":207523491}]},"master_zonegroup":"test-zonegroup","master_zone":"dd308248-8ee8-49be-b928-07c3de7d7aa2","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"7e807acb-79b4-4a09-ba75-06bbedad4839","realm_name":"test-realm","realm_epoch":4} 2021-02-11T09:53:35.457 INFO:rgw_multi.tests:Set sync_from_all flag on zone test-zone2 to false 2021-02-11T09:54:35.458 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=tkscyf-49 2021-02-11T09:54:35.550 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:54:35.551 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:54:35.551 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:54:35.590 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:54:35.590 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:54:35.785 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 2021-02-11T09:54:35.825 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613037275.500429_314.1","last_update":"2021-02-11T09:54:35.500429Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T09:54:35.826 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T09:54:35.826 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:54:35.826 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:54:35.826 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:54:35.867 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:54:35.867 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:54:36.121 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037202.556753_281.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:53:22.556753Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:54:36.122 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613037275.500429_314.1 target=1_1613037202.556753_281.1 2021-02-11T09:54:41.122 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:54:41.122 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:54:41.123 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:54:41.163 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:54:41.163 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:54:41.408 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037202.556753_281.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:53:22.556753Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:54:41.408 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613037275.500429_314.1 target=1_1613037202.556753_281.1 2021-02-11T09:54:46.408 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:54:46.409 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:54:46.409 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:54:46.448 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:54:46.449 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:54:46.685 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037202.556753_281.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:53:22.556753Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:54:46.685 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613037275.500429_314.1 target=1_1613037202.556753_281.1 2021-02-11T09:54:51.686 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:54:51.686 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:54:51.687 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:54:51.729 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:54:51.729 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:54:51.971 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037202.556753_281.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:53:22.556753Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:54:51.972 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613037275.500429_314.1 target=1_1613037202.556753_281.1 2021-02-11T09:54:56.972 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:54:56.973 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:54:56.973 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:54:57.014 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:54:57.014 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:54:57.254 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037202.556753_281.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:53:22.556753Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:54:57.255 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613037275.500429_314.1 target=1_1613037202.556753_281.1 2021-02-11T09:55:02.255 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:55:02.255 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:55:02.256 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:55:02.298 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:55:02.299 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:55:02.561 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037202.556753_281.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:53:22.556753Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:55:02.562 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613037275.500429_314.1 target=1_1613037202.556753_281.1 2021-02-11T09:55:07.562 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:55:07.562 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:55:07.563 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:55:07.605 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:55:07.605 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:55:07.838 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037202.556753_281.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:53:22.556753Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:55:07.838 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613037275.500429_314.1 target=1_1613037202.556753_281.1 2021-02-11T09:55:12.839 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:55:12.839 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:55:12.839 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:55:12.883 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:55:12.883 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:55:13.125 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037202.556753_281.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:53:22.556753Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:55:13.126 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613037275.500429_314.1 target=1_1613037202.556753_281.1 2021-02-11T09:55:18.127 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:55:18.127 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:55:18.128 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:55:18.168 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:55:18.168 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:55:18.408 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037202.556753_281.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:53:22.556753Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:55:18.408 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613037275.500429_314.1 target=1_1613037202.556753_281.1 2021-02-11T09:55:23.409 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:55:23.409 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:55:23.410 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:55:23.448 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:55:23.448 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:55:23.696 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037202.556753_281.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:53:22.556753Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:55:23.697 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613037275.500429_314.1 target=1_1613037202.556753_281.1 2021-02-11T09:55:28.697 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:55:28.697 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:55:28.697 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:55:28.736 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:55:28.736 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:55:28.990 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037202.556753_281.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:53:22.556753Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:55:28.991 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613037275.500429_314.1 target=1_1613037202.556753_281.1 2021-02-11T09:55:33.991 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:55:33.992 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:55:33.992 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:55:34.034 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:55:34.034 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:55:34.271 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037202.556753_281.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:53:22.556753Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:55:34.272 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613037275.500429_314.1 target=1_1613037202.556753_281.1 2021-02-11T09:55:35.671 INFO:tasks.rgw.c1.client.1.smithi094.stdout:2021-02-11T09:55:35.666+0000 7f8fa5ff3700 -1 data sync zone:dd308248 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T09:55:35.674 INFO:tasks.rgw.c2.client.1.smithi098.stdout:2021-02-11T09:55:35.670+0000 7fa8157fa700 -1 data sync zone:dd308248 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T09:55:39.273 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:55:39.273 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:55:39.273 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:55:39.318 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:55:39.318 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:55:39.555 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037275.500429_314.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:54:35.500429Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:55:39.555 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T09:55:39.556 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T09:55:39.556 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:55:39.556 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:55:39.556 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:55:39.596 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:55:39.597 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:55:39.822 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037275.500429_314.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:54:35.500429Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:55:39.823 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T09:55:39.823 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T09:55:39.823 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:55:39.823 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:55:39.823 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:55:39.863 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:55:39.863 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:55:40.096 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037275.500429_314.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:54:35.500429Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:55:40.097 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T09:55:40.114 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=dd308248-8ee8-49be-b928-07c3de7d7aa2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:55:40.114 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=dd308248-8ee8-49be-b928-07c3de7d7aa2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:55:40.115 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --redirect-zone=dd308248-8ee8-49be-b928-07c3de7d7aa2 --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T09:55:40.178 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:55:40.178 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:55:40.207 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:55:40.206+0000 7f93e01c5080 0 failed reading obj info from .rgw.root:zone_info.92d097d8-38ae-4839-9777-3b30db2b2f60: (2) No such file or directory 2021-02-11T09:55:40.208 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:55:40.206+0000 7f93e01c5080 0 WARNING: could not read zone params for zone id=92d097d8-38ae-4839-9777-3b30db2b2f60 name=test-zone3 2021-02-11T09:55:40.208 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:55:40.206+0000 7f93e01c5080 0 failed reading obj info from .rgw.root:zone_info.dd308248-8ee8-49be-b928-07c3de7d7aa2: (2) No such file or directory 2021-02-11T09:55:40.208 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:55:40.206+0000 7f93e01c5080 0 WARNING: could not read zone params for zone id=dd308248-8ee8-49be-b928-07c3de7d7aa2 name=test-zone1 2021-02-11T09:55:40.220 INFO:teuthology.orchestra.run.smithi098.stdout:{"id":"63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"7B4L28ZSM4TSLFS6","secret_key":"eZwNt56BcNJm7h6ZlwjNkhL3ytjGBlGP"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"7e807acb-79b4-4a09-ba75-06bbedad4839","notif_pool":"test-zone2.rgw.log:notif"} 2021-02-11T09:55:40.220 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:55:40.220 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:55:40.220 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T09:55:40.300 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:55:40.300 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:55:40.543 INFO:teuthology.orchestra.run.smithi098.stderr:Sending period to new master zone dd308248-8ee8-49be-b928-07c3de7d7aa2 2021-02-11T09:55:40.649 INFO:teuthology.orchestra.run.smithi098.stdout:{"id":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","epoch":3,"predecessor_uuid":"47caa906-66a7-4be6-bbb9-9f539accb732","sync_status":["1_1613037125.727813_116.1","","",""],"period_map":{"id":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi094.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"dd308248-8ee8-49be-b928-07c3de7d7aa2","zones":[{"id":"45265506-b5d1-4710-9a9e-3e67bee0e451","name":"test-zone4","endpoints":["http://smithi098.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2","name":"test-zone2","endpoints":["http://smithi098.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"false","sync_from":[],"redirect_zone":"dd308248-8ee8-49be-b928-07c3de7d7aa2"},{"id":"92d097d8-38ae-4839-9777-3b30db2b2f60","name":"test-zone3","endpoints":["http://smithi094.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"dd308248-8ee8-49be-b928-07c3de7d7aa2","name":"test-zone1","endpoints":["http://smithi094.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"7e807acb-79b4-4a09-ba75-06bbedad4839","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"45265506-b5d1-4710-9a9e-3e67bee0e451","val":240795374},{"key":"63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2","val":1552258706},{"key":"92d097d8-38ae-4839-9777-3b30db2b2f60","val":2008974102},{"key":"dd308248-8ee8-49be-b928-07c3de7d7aa2","val":207523491}]},"master_zonegroup":"test-zonegroup","master_zone":"dd308248-8ee8-49be-b928-07c3de7d7aa2","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"7e807acb-79b4-4a09-ba75-06bbedad4839","realm_name":"test-realm","realm_epoch":4} 2021-02-11T09:55:40.650 INFO:rgw_multi.tests:Set redirect_zone zone test-zone2 to "dd308248-8ee8-49be-b928-07c3de7d7aa2" 2021-02-11T09:56:40.998 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:56:40.999 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:56:40.999 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --sync-from-all=true --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T09:56:41.040 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:56:41.040 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:56:41.068 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:56:41.065+0000 7f6c3f189080 0 failed reading obj info from .rgw.root:zone_info.92d097d8-38ae-4839-9777-3b30db2b2f60: (2) No such file or directory 2021-02-11T09:56:41.069 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:56:41.065+0000 7f6c3f189080 0 WARNING: could not read zone params for zone id=92d097d8-38ae-4839-9777-3b30db2b2f60 name=test-zone3 2021-02-11T09:56:41.069 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:56:41.065+0000 7f6c3f189080 0 failed reading obj info from .rgw.root:zone_info.dd308248-8ee8-49be-b928-07c3de7d7aa2: (2) No such file or directory 2021-02-11T09:56:41.069 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:56:41.065+0000 7f6c3f189080 0 WARNING: could not read zone params for zone id=dd308248-8ee8-49be-b928-07c3de7d7aa2 name=test-zone1 2021-02-11T09:56:41.081 INFO:teuthology.orchestra.run.smithi098.stdout:{"id":"63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"7B4L28ZSM4TSLFS6","secret_key":"eZwNt56BcNJm7h6ZlwjNkhL3ytjGBlGP"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"7e807acb-79b4-4a09-ba75-06bbedad4839","notif_pool":"test-zone2.rgw.log:notif"} 2021-02-11T09:56:41.082 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:56:41.082 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:56:41.082 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T09:56:41.167 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:56:41.167 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:56:41.426 INFO:teuthology.orchestra.run.smithi098.stderr:Sending period to new master zone dd308248-8ee8-49be-b928-07c3de7d7aa2 2021-02-11T09:56:41.521 INFO:teuthology.orchestra.run.smithi098.stdout:{"id":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","epoch":4,"predecessor_uuid":"47caa906-66a7-4be6-bbb9-9f539accb732","sync_status":["1_1613037125.727813_116.1","","",""],"period_map":{"id":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi094.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"dd308248-8ee8-49be-b928-07c3de7d7aa2","zones":[{"id":"45265506-b5d1-4710-9a9e-3e67bee0e451","name":"test-zone4","endpoints":["http://smithi098.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2","name":"test-zone2","endpoints":["http://smithi098.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":"dd308248-8ee8-49be-b928-07c3de7d7aa2"},{"id":"92d097d8-38ae-4839-9777-3b30db2b2f60","name":"test-zone3","endpoints":["http://smithi094.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"dd308248-8ee8-49be-b928-07c3de7d7aa2","name":"test-zone1","endpoints":["http://smithi094.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"7e807acb-79b4-4a09-ba75-06bbedad4839","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"45265506-b5d1-4710-9a9e-3e67bee0e451","val":240795374},{"key":"63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2","val":1552258706},{"key":"92d097d8-38ae-4839-9777-3b30db2b2f60","val":2008974102},{"key":"dd308248-8ee8-49be-b928-07c3de7d7aa2","val":207523491}]},"master_zonegroup":"test-zonegroup","master_zone":"dd308248-8ee8-49be-b928-07c3de7d7aa2","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"7e807acb-79b4-4a09-ba75-06bbedad4839","realm_name":"test-realm","realm_epoch":4} 2021-02-11T09:56:41.521 INFO:rgw_multi.tests:Set sync_from_all flag on zone test-zone2 to true 2021-02-11T09:57:41.522 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:57:41.522 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:57:41.522 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --redirect-zone= --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T09:57:41.562 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:57:41.563 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:57:41.595 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:57:41.587+0000 7ff17a6a0080 0 failed reading obj info from .rgw.root:zone_info.92d097d8-38ae-4839-9777-3b30db2b2f60: (2) No such file or directory 2021-02-11T09:57:41.595 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:57:41.587+0000 7ff17a6a0080 0 WARNING: could not read zone params for zone id=92d097d8-38ae-4839-9777-3b30db2b2f60 name=test-zone3 2021-02-11T09:57:41.596 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:57:41.591+0000 7ff17a6a0080 0 failed reading obj info from .rgw.root:zone_info.dd308248-8ee8-49be-b928-07c3de7d7aa2: (2) No such file or directory 2021-02-11T09:57:41.596 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:57:41.591+0000 7ff17a6a0080 0 WARNING: could not read zone params for zone id=dd308248-8ee8-49be-b928-07c3de7d7aa2 name=test-zone1 2021-02-11T09:57:41.610 INFO:teuthology.orchestra.run.smithi098.stdout:{"id":"63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"7B4L28ZSM4TSLFS6","secret_key":"eZwNt56BcNJm7h6ZlwjNkhL3ytjGBlGP"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"7e807acb-79b4-4a09-ba75-06bbedad4839","notif_pool":"test-zone2.rgw.log:notif"} 2021-02-11T09:57:41.610 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:57:41.610 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:57:41.611 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T09:57:41.692 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:57:41.692 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:57:41.927 INFO:teuthology.orchestra.run.smithi098.stderr:Sending period to new master zone dd308248-8ee8-49be-b928-07c3de7d7aa2 2021-02-11T09:57:42.023 INFO:teuthology.orchestra.run.smithi098.stdout:{"id":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","epoch":5,"predecessor_uuid":"47caa906-66a7-4be6-bbb9-9f539accb732","sync_status":["1_1613037125.727813_116.1","","",""],"period_map":{"id":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi094.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"dd308248-8ee8-49be-b928-07c3de7d7aa2","zones":[{"id":"45265506-b5d1-4710-9a9e-3e67bee0e451","name":"test-zone4","endpoints":["http://smithi098.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2","name":"test-zone2","endpoints":["http://smithi098.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"92d097d8-38ae-4839-9777-3b30db2b2f60","name":"test-zone3","endpoints":["http://smithi094.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"dd308248-8ee8-49be-b928-07c3de7d7aa2","name":"test-zone1","endpoints":["http://smithi094.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"7e807acb-79b4-4a09-ba75-06bbedad4839","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"45265506-b5d1-4710-9a9e-3e67bee0e451","val":240795374},{"key":"63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2","val":1552258706},{"key":"92d097d8-38ae-4839-9777-3b30db2b2f60","val":2008974102},{"key":"dd308248-8ee8-49be-b928-07c3de7d7aa2","val":207523491}]},"master_zonegroup":"test-zonegroup","master_zone":"dd308248-8ee8-49be-b928-07c3de7d7aa2","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"7e807acb-79b4-4a09-ba75-06bbedad4839","realm_name":"test-realm","realm_epoch":4} 2021-02-11T09:57:42.023 INFO:rgw_multi.tests:Set redirect_zone zone test-zone2 to "" 2021-02-11T09:58:42.025 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_multi_zone_redirect ... ok 2021-02-11T09:58:42.026 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:58:42.026 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:58:42.027 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:58:42.068 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:58:42.069 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:58:42.267 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 2021-02-11T09:58:42.302 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613037275.500429_314.1","last_update":"2021-02-11T09:54:35.500429Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T09:58:42.303 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T09:58:42.303 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:58:42.303 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:58:42.303 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:58:42.341 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:58:42.341 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:58:42.577 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037275.500429_314.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:54:35.500429Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:58:42.578 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T09:58:42.578 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T09:58:42.578 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:58:42.578 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:58:42.579 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:58:42.619 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:58:42.619 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:58:42.851 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037275.500429_314.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:54:35.500429Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:58:42.851 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T09:58:42.851 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T09:58:42.851 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:58:42.851 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:58:42.851 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:58:42.891 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:58:42.891 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:58:43.125 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037275.500429_314.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:54:35.500429Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:58:43.126 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T09:58:43.126 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', '7B4L28ZSM4TSLFS6', '--secret', 'eZwNt56BcNJm7h6ZlwjNkhL3ytjGBlGP', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:58:43.126 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', '7B4L28ZSM4TSLFS6', '--secret', 'eZwNt56BcNJm7h6ZlwjNkhL3ytjGBlGP', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:58:43.126 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key 7B4L28ZSM4TSLFS6 --secret eZwNt56BcNJm7h6ZlwjNkhL3ytjGBlGP --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T09:58:43.209 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:58:43.209 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:58:43.277 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:58:43.274+0000 7f8f6bbc6080 0 failed reading obj info from .rgw.root:zone_info.92d097d8-38ae-4839-9777-3b30db2b2f60: (2) No such file or directory 2021-02-11T09:58:43.277 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:58:43.274+0000 7f8f6bbc6080 0 WARNING: could not read zone params for zone id=92d097d8-38ae-4839-9777-3b30db2b2f60 name=test-zone3 2021-02-11T09:58:43.279 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:58:43.274+0000 7f8f6bbc6080 0 failed reading obj info from .rgw.root:zone_info.dd308248-8ee8-49be-b928-07c3de7d7aa2: (2) No such file or directory 2021-02-11T09:58:43.279 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:58:43.274+0000 7f8f6bbc6080 0 WARNING: could not read zone params for zone id=dd308248-8ee8-49be-b928-07c3de7d7aa2 name=test-zone1 2021-02-11T09:58:43.291 INFO:teuthology.orchestra.run.smithi098.stdout:{"id":"d943390e-5685-4658-9d6d-486d68d2fc4d","name":"remove","domain_root":"remove.rgw.meta:root","control_pool":"remove.rgw.control","gc_pool":"remove.rgw.log:gc","lc_pool":"remove.rgw.log:lc","log_pool":"remove.rgw.log","intent_log_pool":"remove.rgw.log:intent","usage_log_pool":"remove.rgw.log:usage","roles_pool":"remove.rgw.meta:roles","reshard_pool":"remove.rgw.log:reshard","user_keys_pool":"remove.rgw.meta:users.keys","user_email_pool":"remove.rgw.meta:users.email","user_swift_pool":"remove.rgw.meta:users.swift","user_uid_pool":"remove.rgw.meta:users.uid","otp_pool":"remove.rgw.otp","system_key":{"access_key":"7B4L28ZSM4TSLFS6","secret_key":"eZwNt56BcNJm7h6ZlwjNkhL3ytjGBlGP"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"remove.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"remove.rgw.buckets.data"}},"data_extra_pool":"remove.rgw.buckets.non-ec","index_type":0}}],"realm_id":"7e807acb-79b4-4a09-ba75-06bbedad4839","notif_pool":"remove.rgw.log:notif"} 2021-02-11T09:58:43.291 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:58:43.291 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:58:43.291 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T09:58:43.374 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:58:43.375 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:58:50.717 INFO:teuthology.orchestra.run.smithi098.stderr:Sending period to new master zone dd308248-8ee8-49be-b928-07c3de7d7aa2 2021-02-11T09:58:50.815 INFO:teuthology.orchestra.run.smithi098.stdout:{"id":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","epoch":6,"predecessor_uuid":"47caa906-66a7-4be6-bbb9-9f539accb732","sync_status":["1_1613037125.727813_116.1","","",""],"period_map":{"id":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi094.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"dd308248-8ee8-49be-b928-07c3de7d7aa2","zones":[{"id":"45265506-b5d1-4710-9a9e-3e67bee0e451","name":"test-zone4","endpoints":["http://smithi098.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2","name":"test-zone2","endpoints":["http://smithi098.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"92d097d8-38ae-4839-9777-3b30db2b2f60","name":"test-zone3","endpoints":["http://smithi094.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"d943390e-5685-4658-9d6d-486d68d2fc4d","name":"remove","endpoints":[],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"dd308248-8ee8-49be-b928-07c3de7d7aa2","name":"test-zone1","endpoints":["http://smithi094.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"7e807acb-79b4-4a09-ba75-06bbedad4839","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"45265506-b5d1-4710-9a9e-3e67bee0e451","val":240795374},{"key":"63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2","val":1552258706},{"key":"92d097d8-38ae-4839-9777-3b30db2b2f60","val":2008974102},{"key":"d943390e-5685-4658-9d6d-486d68d2fc4d","val":1407206174},{"key":"dd308248-8ee8-49be-b928-07c3de7d7aa2","val":207523491}]},"master_zonegroup":"test-zonegroup","master_zone":"dd308248-8ee8-49be-b928-07c3de7d7aa2","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"7e807acb-79b4-4a09-ba75-06bbedad4839","realm_name":"test-realm","realm_epoch":4} 2021-02-11T09:58:50.815 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:58:50.816 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:58:50.816 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T09:58:50.854 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:58:50.854 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:58:50.876 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:58:50.870+0000 7fdfa1e80080 0 failed reading obj info from .rgw.root:zone_info.45265506-b5d1-4710-9a9e-3e67bee0e451: (2) No such file or directory 2021-02-11T09:58:50.876 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:58:50.870+0000 7fdfa1e80080 0 WARNING: could not read zone params for zone id=45265506-b5d1-4710-9a9e-3e67bee0e451 name=test-zone4 2021-02-11T09:58:50.877 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:58:50.870+0000 7fdfa1e80080 0 failed reading obj info from .rgw.root:zone_info.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2: (2) No such file or directory 2021-02-11T09:58:50.877 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:58:50.870+0000 7fdfa1e80080 0 WARNING: could not read zone params for zone id=63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2 name=test-zone2 2021-02-11T09:58:50.885 INFO:teuthology.orchestra.run.smithi094.stdout:{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi094.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"dd308248-8ee8-49be-b928-07c3de7d7aa2","zones":[{"id":"45265506-b5d1-4710-9a9e-3e67bee0e451","name":"test-zone4","endpoints":["http://smithi098.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2","name":"test-zone2","endpoints":["http://smithi098.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"92d097d8-38ae-4839-9777-3b30db2b2f60","name":"test-zone3","endpoints":["http://smithi094.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"dd308248-8ee8-49be-b928-07c3de7d7aa2","name":"test-zone1","endpoints":["http://smithi094.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"7e807acb-79b4-4a09-ba75-06bbedad4839","sync_policy":{"groups":[]}} 2021-02-11T09:58:50.885 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:58:50.885 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:58:50.885 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T09:58:50.967 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:58:50.967 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:58:50.997 INFO:teuthology.orchestra.run.smithi094.stderr:zone name remove not found in zonegroup test-zonegroup 2021-02-11T09:58:51.004 DEBUG:teuthology.orchestra.run:got remote process result: 2 2021-02-11T09:58:51.005 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:58:51.005 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:58:51.005 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone delete --rgw-zone remove --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T09:58:51.044 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:58:51.044 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:58:51.090 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:58:51.085+0000 7f396568e080 0 failed reading obj info from .rgw.root:zone_info.92d097d8-38ae-4839-9777-3b30db2b2f60: (2) No such file or directory 2021-02-11T09:58:51.090 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:58:51.085+0000 7f396568e080 0 WARNING: could not read zone params for zone id=92d097d8-38ae-4839-9777-3b30db2b2f60 name=test-zone3 2021-02-11T09:58:51.090 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:58:51.085+0000 7f396568e080 0 failed reading obj info from .rgw.root:zone_info.dd308248-8ee8-49be-b928-07c3de7d7aa2: (2) No such file or directory 2021-02-11T09:58:51.090 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:58:51.085+0000 7f396568e080 0 WARNING: could not read zone params for zone id=dd308248-8ee8-49be-b928-07c3de7d7aa2 name=test-zone1 2021-02-11T09:58:51.097 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T09:58:51.093+0000 7f396568e080 0 zone id d943390e-5685-4658-9d6d-486d68d2fc4d is not a part of zonegroup default 2021-02-11T09:58:51.111 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:58:51.111 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:58:51.111 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T09:58:51.159 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:58:51.159 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:58:51.224 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:58:51.218+0000 7f9eb696b080 0 WARNING: can't generate connection for zone d943390e-5685-4658-9d6d-486d68d2fc4d id remove: no endpoints defined 2021-02-11T09:58:51.409 INFO:teuthology.orchestra.run.smithi094.stdout:{"id":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","epoch":7,"predecessor_uuid":"47caa906-66a7-4be6-bbb9-9f539accb732","sync_status":["1_1613037125.727813_116.1","","",""],"period_map":{"id":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi094.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"dd308248-8ee8-49be-b928-07c3de7d7aa2","zones":[{"id":"45265506-b5d1-4710-9a9e-3e67bee0e451","name":"test-zone4","endpoints":["http://smithi098.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2","name":"test-zone2","endpoints":["http://smithi098.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"92d097d8-38ae-4839-9777-3b30db2b2f60","name":"test-zone3","endpoints":["http://smithi094.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"dd308248-8ee8-49be-b928-07c3de7d7aa2","name":"test-zone1","endpoints":["http://smithi094.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"7e807acb-79b4-4a09-ba75-06bbedad4839","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"45265506-b5d1-4710-9a9e-3e67bee0e451","val":240795374},{"key":"63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2","val":1552258706},{"key":"92d097d8-38ae-4839-9777-3b30db2b2f60","val":2008974102},{"key":"dd308248-8ee8-49be-b928-07c3de7d7aa2","val":207523491}]},"master_zonegroup":"test-zonegroup","master_zone":"dd308248-8ee8-49be-b928-07c3de7d7aa2","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"7e807acb-79b4-4a09-ba75-06bbedad4839","realm_name":"test-realm","realm_epoch":4} 2021-02-11T09:58:51.410 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_zonegroup_remove ... ok 2021-02-11T09:58:51.410 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:58:51.410 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:58:51.411 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup create --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T09:58:51.447 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:58:51.447 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:58:51.475 INFO:teuthology.orchestra.run.smithi094.stdout:{"id":"bd3ce122-2fef-434d-b171-c4f6191b86e3","name":"remove_zg","api_name":"remove_zg","is_master":"false","endpoints":[],"hostnames":[],"hostnames_s3website":[],"master_zone":"","zones":[],"placement_targets":[],"default_placement":"","realm_id":"7e807acb-79b4-4a09-ba75-06bbedad4839","sync_policy":{"groups":[]}} 2021-02-11T09:58:51.475 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:58:51.475 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:58:51.475 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T09:58:51.556 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:58:51.556 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:58:51.610 INFO:teuthology.orchestra.run.smithi094.stdout:{"id":"8755df2e-e08a-4d76-8b29-d344b9fb28b5","name":"remove","domain_root":"remove.rgw.meta:root","control_pool":"remove.rgw.control","gc_pool":"remove.rgw.log:gc","lc_pool":"remove.rgw.log:lc","log_pool":"remove.rgw.log","intent_log_pool":"remove.rgw.log:intent","usage_log_pool":"remove.rgw.log:usage","roles_pool":"remove.rgw.meta:roles","reshard_pool":"remove.rgw.log:reshard","user_keys_pool":"remove.rgw.meta:users.keys","user_email_pool":"remove.rgw.meta:users.email","user_swift_pool":"remove.rgw.meta:users.swift","user_uid_pool":"remove.rgw.meta:users.uid","otp_pool":"remove.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"remove.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"remove.rgw.buckets.data"}},"data_extra_pool":"remove.rgw.buckets.non-ec","index_type":0}}],"realm_id":"7e807acb-79b4-4a09-ba75-06bbedad4839","notif_pool":"remove.rgw.log:notif"} 2021-02-11T09:58:51.611 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:58:51.611 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:58:51.611 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T09:58:51.696 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:58:51.696 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:58:51.944 INFO:teuthology.orchestra.run.smithi094.stdout:{"id":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","epoch":8,"predecessor_uuid":"47caa906-66a7-4be6-bbb9-9f539accb732","sync_status":["1_1613037125.727813_116.1","","",""],"period_map":{"id":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","zonegroups":[{"id":"bd3ce122-2fef-434d-b171-c4f6191b86e3","name":"remove_zg","api_name":"remove_zg","is_master":"false","endpoints":[],"hostnames":[],"hostnames_s3website":[],"master_zone":"8755df2e-e08a-4d76-8b29-d344b9fb28b5","zones":[{"id":"8755df2e-e08a-4d76-8b29-d344b9fb28b5","name":"remove","endpoints":[],"log_meta":"false","log_data":"false","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"7e807acb-79b4-4a09-ba75-06bbedad4839","sync_policy":{"groups":[]}},{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi094.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"dd308248-8ee8-49be-b928-07c3de7d7aa2","zones":[{"id":"45265506-b5d1-4710-9a9e-3e67bee0e451","name":"test-zone4","endpoints":["http://smithi098.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2","name":"test-zone2","endpoints":["http://smithi098.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"92d097d8-38ae-4839-9777-3b30db2b2f60","name":"test-zone3","endpoints":["http://smithi094.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"dd308248-8ee8-49be-b928-07c3de7d7aa2","name":"test-zone1","endpoints":["http://smithi094.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"7e807acb-79b4-4a09-ba75-06bbedad4839","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"45265506-b5d1-4710-9a9e-3e67bee0e451","val":240795374},{"key":"63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2","val":1552258706},{"key":"8755df2e-e08a-4d76-8b29-d344b9fb28b5","val":658735321},{"key":"92d097d8-38ae-4839-9777-3b30db2b2f60","val":2008974102},{"key":"dd308248-8ee8-49be-b928-07c3de7d7aa2","val":207523491}]},"master_zonegroup":"test-zonegroup","master_zone":"dd308248-8ee8-49be-b928-07c3de7d7aa2","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"7e807acb-79b4-4a09-ba75-06bbedad4839","realm_name":"test-realm","realm_epoch":4} 2021-02-11T09:58:51.944 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:58:51.944 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:58:51.945 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone delete --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T09:58:52.025 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:58:52.025 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:58:52.072 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:58:52.066+0000 7f5ee90d6080 0 zone id 8755df2e-e08a-4d76-8b29-d344b9fb28b5 is not a part of zonegroup test-zonegroup 2021-02-11T09:58:52.073 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:58:52.066+0000 7f5ee90d6080 0 zone id 8755df2e-e08a-4d76-8b29-d344b9fb28b5 is not a part of zonegroup default 2021-02-11T09:58:52.084 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:58:52.084 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:58:52.084 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T09:58:52.165 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:58:52.165 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:58:52.206 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T09:58:52.198+0000 7f17bbf49080 0 ERROR: zonegroup remove_zg has a non existent master zone 2021-02-11T09:58:52.212 DEBUG:teuthology.orchestra.run:got remote process result: 22 2021-02-11T09:58:52.212 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:58:52.212 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:58:52.212 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup delete --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T09:58:52.289 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:58:52.290 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:58:52.320 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:58:52.320 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T09:58:52.321 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T09:58:52.401 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:58:52.401 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:58:52.632 INFO:teuthology.orchestra.run.smithi094.stdout:{"id":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","epoch":9,"predecessor_uuid":"47caa906-66a7-4be6-bbb9-9f539accb732","sync_status":["1_1613037125.727813_116.1","","",""],"period_map":{"id":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi094.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"dd308248-8ee8-49be-b928-07c3de7d7aa2","zones":[{"id":"45265506-b5d1-4710-9a9e-3e67bee0e451","name":"test-zone4","endpoints":["http://smithi098.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2","name":"test-zone2","endpoints":["http://smithi098.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"92d097d8-38ae-4839-9777-3b30db2b2f60","name":"test-zone3","endpoints":["http://smithi094.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"dd308248-8ee8-49be-b928-07c3de7d7aa2","name":"test-zone1","endpoints":["http://smithi094.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"7e807acb-79b4-4a09-ba75-06bbedad4839","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"45265506-b5d1-4710-9a9e-3e67bee0e451","val":240795374},{"key":"63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2","val":1552258706},{"key":"92d097d8-38ae-4839-9777-3b30db2b2f60","val":2008974102},{"key":"dd308248-8ee8-49be-b928-07c3de7d7aa2","val":207523491}]},"master_zonegroup":"test-zonegroup","master_zone":"dd308248-8ee8-49be-b928-07c3de7d7aa2","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"7e807acb-79b4-4a09-ba75-06bbedad4839","realm_name":"test-realm","realm_epoch":4} 2021-02-11T09:58:52.633 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_zg_master_zone_delete ... ok 2021-02-11T09:58:52.633 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=tkscyf-50 2021-02-11T09:58:52.953 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=tkscyf-51 2021-02-11T09:58:53.612 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=tkscyf-52 2021-02-11T09:58:53.676 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=tkscyf-53 2021-02-11T09:58:53.708 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=tkscyf-54 2021-02-11T09:58:53.782 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=tkscyf-55 2021-02-11T09:58:56.042 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_set_bucket_website ... tasks.rgw_multi.tests.test_set_bucket_policy ... ok 2021-02-11T09:58:56.043 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=tkscyf-56 2021-02-11T09:58:56.082 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=tkscyf-57 2021-02-11T09:58:56.157 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=tkscyf-58 2021-02-11T09:58:56.215 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:58:56.215 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:58:56.215 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:58:56.248 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:58:56.248 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:58:56.428 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 2021-02-11T09:58:56.464 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613037536.194388_466.1","last_update":"2021-02-11T09:58:56.194388Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T09:58:56.464 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T09:58:56.464 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:58:56.464 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:58:56.465 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:58:56.504 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:58:56.504 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:58:56.750 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037275.500429_314.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:54:35.500429Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:58:56.751 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613037536.194388_466.1 target=1_1613037275.500429_314.1 2021-02-11T09:59:01.751 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:59:01.752 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:59:01.752 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:59:01.791 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:59:01.791 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:59:02.048 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037275.500429_314.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:54:35.500429Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:59:02.048 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613037536.194388_466.1 target=1_1613037275.500429_314.1 2021-02-11T09:59:07.049 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:59:07.049 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:59:07.050 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:59:07.092 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:59:07.092 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:59:07.345 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037275.500429_314.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:54:35.500429Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:59:07.345 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613037536.194388_466.1 target=1_1613037275.500429_314.1 2021-02-11T09:59:12.346 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:59:12.346 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:59:12.347 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:59:12.387 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:59:12.387 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:59:12.649 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037275.500429_314.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:54:35.500429Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:59:12.649 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613037536.194388_466.1 target=1_1613037275.500429_314.1 2021-02-11T09:59:17.650 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:59:17.650 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:59:17.650 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:59:17.691 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:59:17.691 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:59:17.933 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037275.500429_314.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:54:35.500429Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:59:17.933 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613037536.194388_466.1 target=1_1613037275.500429_314.1 2021-02-11T09:59:22.934 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:59:22.935 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:59:22.935 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:59:22.974 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:59:22.974 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:59:23.212 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037275.500429_314.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:54:35.500429Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:59:23.213 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613037536.194388_466.1 target=1_1613037275.500429_314.1 2021-02-11T09:59:28.213 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:59:28.213 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:59:28.214 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:59:28.255 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:59:28.256 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:59:28.503 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037275.500429_314.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:54:35.500429Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:59:28.504 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613037536.194388_466.1 target=1_1613037275.500429_314.1 2021-02-11T09:59:33.505 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:59:33.505 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:59:33.506 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:59:33.546 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:59:33.546 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:59:33.815 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037275.500429_314.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:54:35.500429Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:59:33.815 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613037536.194388_466.1 target=1_1613037275.500429_314.1 2021-02-11T09:59:38.816 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:59:38.816 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:59:38.816 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:59:38.857 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:59:38.858 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:59:39.104 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037275.500429_314.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:54:35.500429Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:59:39.104 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613037536.194388_466.1 target=1_1613037275.500429_314.1 2021-02-11T09:59:44.105 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:59:44.105 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:59:44.106 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:59:44.148 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:59:44.148 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:59:44.394 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037275.500429_314.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:54:35.500429Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:59:44.394 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613037536.194388_466.1 target=1_1613037275.500429_314.1 2021-02-11T09:59:49.396 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:59:49.396 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:59:49.396 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:59:49.437 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:59:49.437 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:59:49.674 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037275.500429_314.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:54:35.500429Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:59:49.674 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613037536.194388_466.1 target=1_1613037275.500429_314.1 2021-02-11T09:59:54.675 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:59:54.675 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:59:54.676 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:59:54.715 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:59:54.715 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T09:59:54.951 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037275.500429_314.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:54:35.500429Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T09:59:54.951 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613037536.194388_466.1 target=1_1613037275.500429_314.1 2021-02-11T09:59:59.952 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:59:59.952 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T09:59:59.953 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T09:59:59.994 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T09:59:59.994 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:00:00.248 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037275.500429_314.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:54:35.500429Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:00:00.248 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613037536.194388_466.1 target=1_1613037275.500429_314.1 2021-02-11T10:00:05.249 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:00:05.249 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:00:05.249 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:00:05.291 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:00:05.292 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:00:05.518 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037275.500429_314.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:54:35.500429Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:00:05.519 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613037536.194388_466.1 target=1_1613037275.500429_314.1 2021-02-11T10:00:10.520 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:00:10.520 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:00:10.520 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:00:10.559 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:00:10.559 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:00:10.810 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037275.500429_314.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:54:35.500429Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:00:10.810 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613037536.194388_466.1 target=1_1613037275.500429_314.1 2021-02-11T10:00:15.811 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:00:15.811 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:00:15.811 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:00:15.851 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:00:15.851 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:00:16.112 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037275.500429_314.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:54:35.500429Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:00:16.113 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613037536.194388_466.1 target=1_1613037275.500429_314.1 2021-02-11T10:00:21.114 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:00:21.114 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:00:21.114 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:00:21.156 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:00:21.156 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:00:21.415 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037275.500429_314.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:54:35.500429Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:00:21.416 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613037536.194388_466.1 target=1_1613037275.500429_314.1 2021-02-11T10:00:26.417 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:00:32.779 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:00:32.779 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:00:32.822 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:00:32.822 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:00:33.066 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037275.500429_314.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:54:35.500429Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:00:33.066 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613037536.194388_466.1 target=1_1613037275.500429_314.1 2021-02-11T10:00:38.067 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:00:38.068 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:00:38.068 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:00:38.109 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:00:38.109 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:00:38.372 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037275.500429_314.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:54:35.500429Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:00:38.373 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613037536.194388_466.1 target=1_1613037275.500429_314.1 2021-02-11T10:00:43.374 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:00:43.374 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:00:43.375 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:00:43.413 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:00:43.414 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:00:43.662 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037275.500429_314.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:54:35.500429Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:00:43.662 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613037536.194388_466.1 target=1_1613037275.500429_314.1 2021-02-11T10:00:48.663 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:00:48.663 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:00:48.664 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:00:48.703 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:00:48.703 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:00:48.949 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037275.500429_314.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:54:35.500429Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:00:48.949 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613037536.194388_466.1 target=1_1613037275.500429_314.1 2021-02-11T10:00:53.950 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:00:53.950 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:00:53.950 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:00:53.992 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:00:53.992 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:00:54.246 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037275.500429_314.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:54:35.500429Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:00:54.247 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613037536.194388_466.1 target=1_1613037275.500429_314.1 2021-02-11T10:00:59.248 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:00:59.248 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:00:59.248 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:00:59.288 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:00:59.288 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:00:59.530 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037536.194388_466.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:58:56.194388Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:00:59.531 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T10:00:59.532 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T10:00:59.532 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:00:59.532 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:00:59.532 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:00:59.573 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:00:59.573 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:00:59.804 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037536.194388_466.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:58:56.194388Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:00:59.804 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T10:00:59.805 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:00:59.805 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:00:59.805 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:00:59.805 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:00:59.845 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:00:59.845 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:01:00.074 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037536.194388_466.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T09:58:56.194388Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:01:00.074 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:01:00.074 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'tkscyf-56', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:01:00.075 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'tkscyf-56', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:01:00.075 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket tkscyf-56 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T10:01:00.116 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:01:00.116 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:01:00.414 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'tkscyf-57', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:01:00.414 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'tkscyf-57', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:01:00.414 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket tkscyf-57 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T10:01:00.496 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:01:00.497 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:01:00.785 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'tkscyf-58', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:01:00.786 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'tkscyf-58', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:01:00.786 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket tkscyf-58 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T10:01:00.866 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:01:00.866 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:01:01.171 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:01:01.172 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:01:01.172 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:01:01.252 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:01:01.253 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:01:01.486 INFO:teuthology.orchestra.run.smithi094.stdout:["obj_delete_at_hint.0000000078","meta.history","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-10:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.10:8","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-16:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.16:5","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.37-tkscyf-37","obj_delete_at_hint.0000000070","obj_delete_at_hint.0000000104","obj_delete_at_hint.0000000026","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.22-tkscyf-22","2021-02-11-09-63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2.5092.2-tkscyf-43","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-26:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.26:8","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-11:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.11:7","obj_delete_at_hint.0000000028","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-26:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.26:8","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-48:dd308248-8ee8-49be-b928-07c3de7d7aa2.5145.5:8","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.15-tkscyf-15","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-11:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.11:7","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.26-tkscyf-26","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.5352.1-tkscyf-49","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-12:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.12:7","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.17-tkscyf-17","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-37:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.37:10","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.40-tkscyf-40","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.6-tkscyf-6","obj_delete_at_hint.0000000040","meta.full-sync.index.0","obj_delete_at_hint.0000000015","obj_delete_at_hint.0000000069","obj_delete_at_hint.0000000095","obj_delete_at_hint.0000000003","obj_delete_at_hint.0000000047","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-10:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.10:8","obj_delete_at_hint.0000000035","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-24:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.24:8","obj_delete_at_hint.0000000037","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.31-tkscyf-31","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-16:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.16:1","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.5145.4-tkscyf-47","obj_delete_at_hint.0000000024","obj_delete_at_hint.0000000000","obj_delete_at_hint.0000000031","data_log.3","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-10:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.10:3","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.5145.3-tkscyf-46","data_loggenerations_metadata","2021-02-11-09-63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2.5092.1-tkscyf-42","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-49:dd308248-8ee8-49be-b928-07c3de7d7aa2.5352.1:5","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-23:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.23:8","obj_delete_at_hint.0000000076","obj_delete_at_hint.0000000113","obj_delete_at_hint.0000000005","obj_delete_at_hint.0000000011","obj_delete_at_hint.0000000058","obj_delete_at_hint.0000000052","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.19-tkscyf-19","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.29-tkscyf-29","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-11:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.11:8","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-12:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.12:3","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-18:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.18:5","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-25:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.25:8","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.5145.5-tkscyf-48","obj_delete_at_hint.0000000088","obj_delete_at_hint.0000000080","obj_delete_at_hint.0000000090","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.23-tkscyf-23","obj_delete_at_hint.0000000110","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.13-tkscyf-13","obj_delete_at_hint.0000000096","obj_delete_at_hint.0000000087","obj_delete_at_hint.0000000008","obj_delete_at_hint.0000000006","obj_delete_at_hint.0000000029","obj_delete_at_hint.0000000089","obj_delete_at_hint.0000000022","meta.log.0d37c290-12ce-40b6-8dd3-c3f1d036f1a7.0","bilog.trim","obj_delete_at_hint.0000000016","obj_delete_at_hint.0000000041","data.full-sync.index.92d097d8-38ae-4839-9777-3b30db2b2f60.0","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-12:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.12:8","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.11-tkscyf-11","obj_delete_at_hint.0000000018","obj_delete_at_hint.0000000092","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-10:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.10:7","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-17:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.17:5","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-10:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.10:7","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-13:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.13:1","obj_delete_at_hint.0000000014","obj_delete_at_hint.0000000112","obj_delete_at_hint.0000000007","obj_delete_at_hint.0000000021","obj_delete_at_hint.0000000064","obj_delete_at_hint.0000000071","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.5-tkscyf-5","obj_delete_at_hint.0000000074","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-18:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.18:5","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-17:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.17:5","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.32-tkscyf-32","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.3-tkscyf-52","obj_delete_at_hint.0000000081","obj_delete_at_hint.0000000009","data.full-sync.index.92d097d8-38ae-4839-9777-3b30db2b2f60.2","obj_delete_at_hint.0000000121","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.9-tkscyf-9","data_log.1","obj_delete_at_hint.0000000125","obj_delete_at_hint.0000000082","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-16:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.16:5","meta.full-sync.index.1","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-37:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.37:10","obj_delete_at_hint.0000000105","obj_delete_at_hint.0000000059","datalog.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.12-tkscyf-12","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.8-tkscyf-8","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.4-tkscyf-53","obj_delete_at_hint.0000000077","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-18:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.18:3","obj_delete_at_hint.0000000032","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-12:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.12:7","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-38:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.38:10","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.9-tkscyf-58","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-12:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.12:1","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.25-tkscyf-25","obj_delete_at_hint.0000000053","obj_delete_at_hint.0000000091","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.3-tkscyf-3","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.5145.1-tkscyf-44","obj_delete_at_hint.0000000065","obj_delete_at_hint.0000000083","obj_delete_at_hint.0000000010","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-20:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.20:8","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-10:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.10:3","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-36:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.36:10","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.41-tkscyf-41","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-11:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.11:1","mdlog.sync-status.shard.2","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.24-tkscyf-24","data.full-sync.index.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2.1","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.35-tkscyf-35","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.7-tkscyf-56","obj_delete_at_hint.0000000045","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-22:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.22:8","obj_delete_at_hint.0000000002","obj_delete_at_hint.0000000116","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-18:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.18:1","obj_delete_at_hint.0000000034","obj_delete_at_hint.0000000101","mdlog.sync-status.shard.1","obj_delete_at_hint.0000000079","obj_delete_at_hint.0000000049","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-16:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.16:3","obj_delete_at_hint.0000000117","obj_delete_at_hint.0000000044","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-38:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.38:10","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-12:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.12:8","obj_delete_at_hint.0000000066","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.27-tkscyf-27","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.33-tkscyf-33","obj_delete_at_hint.0000000068","obj_delete_at_hint.0000000085","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.18-tkscyf-18","obj_delete_at_hint.0000000073","obj_delete_at_hint.0000000038","datalog.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60","meta.full-sync.index.3","obj_delete_at_hint.0000000118","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.1-tkscyf-50","obj_delete_at_hint.0000000036","obj_delete_at_hint.0000000103","mdlog.sync-status.shard.3","obj_delete_at_hint.0000000119","obj_delete_at_hint.0000000098","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.34-tkscyf-34","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.5145.2-tkscyf-45","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-18:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.18:1","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.30-tkscyf-30","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.38-tkscyf-38","mdlog.sync-status.shard.0","obj_delete_at_hint.0000000027","data.full-sync.index.92d097d8-38ae-4839-9777-3b30db2b2f60.3","mdlog.sync-status","obj_delete_at_hint.0000000019","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-13:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.13:1","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.6-tkscyf-55","obj_delete_at_hint.0000000039","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-12:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.12:3","datalog.sync-status.shard.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2.1","obj_delete_at_hint.0000000100","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.14-tkscyf-14","obj_delete_at_hint.0000000093","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-17:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.17:3","obj_delete_at_hint.0000000004","obj_delete_at_hint.0000000063","obj_delete_at_hint.0000000122","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-25:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.25:8","obj_delete_at_hint.0000000057","datalog.sync-status.shard.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2.2","obj_delete_at_hint.0000000054","obj_delete_at_hint.0000000114","data_log.0","obj_delete_at_hint.0000000012","data.full-sync.index.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2.0","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-46:dd308248-8ee8-49be-b928-07c3de7d7aa2.5145.3:8","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.10-tkscyf-10","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.1-tkscyf-1","obj_delete_at_hint.0000000084","obj_delete_at_hint.0000000043","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-17:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.17:3","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-17:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.17:1","obj_delete_at_hint.0000000111","datalog.sync-status.shard.92d097d8-38ae-4839-9777-3b30db2b2f60.1","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-19:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.19:8","obj_delete_at_hint.0000000048","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.2-tkscyf-51","obj_delete_at_hint.0000000020","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-10:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.10:1","datalog.sync-status.shard.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2.0","data.full-sync.index.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2.2","obj_delete_at_hint.0000000099","obj_delete_at_hint.0000000056","data.full-sync.index.92d097d8-38ae-4839-9777-3b30db2b2f60.1","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-10:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.10:1","obj_delete_at_hint.0000000072","obj_delete_at_hint.0000000062","obj_delete_at_hint.0000000115","obj_delete_at_hint.0000000033","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-15:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.15:1","data.full-sync.index.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2.3","obj_delete_at_hint.0000000025","obj_delete_at_hint.0000000123","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.21-tkscyf-21","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-19:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.19:5","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-14:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.14:1","obj_delete_at_hint.0000000108","obj_delete_at_hint.0000000094","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.16-tkscyf-16","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-47:dd308248-8ee8-49be-b928-07c3de7d7aa2.5145.4:8","obj_delete_at_hint.0000000060","obj_delete_at_hint.0000000109","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-46:dd308248-8ee8-49be-b928-07c3de7d7aa2.5145.3:8","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-14:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.14:1","obj_delete_at_hint.0000000013","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-19:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.19:5","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-16:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.16:1","obj_delete_at_hint.0000000051","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-36:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.36:10","obj_delete_at_hint.0000000106","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.39-tkscyf-39","obj_delete_at_hint.0000000017","obj_delete_at_hint.0000000023","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.5-tkscyf-54","obj_delete_at_hint.0000000107","obj_delete_at_hint.0000000046","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-11:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.11:8","meta.full-sync.index.2","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-19:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.19:8","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-24:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.24:8","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-11:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.11:3","data_log.2","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-17:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.17:1","obj_delete_at_hint.0000000067","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.8-tkscyf-57","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.2-tkscyf-2","obj_delete_at_hint.0000000086","datalog.sync-status.shard.92d097d8-38ae-4839-9777-3b30db2b2f60.0","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.7-tkscyf-7","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-22:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.22:8","datalog.sync-status.shard.92d097d8-38ae-4839-9777-3b30db2b2f60.3","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-12:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.12:1","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-21:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.21:8","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-47:dd308248-8ee8-49be-b928-07c3de7d7aa2.5145.4:8","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-11:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.11:3","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.28-tkscyf-28","obj_delete_at_hint.0000000050","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-11:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.11:1","obj_delete_at_hint.0000000102","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-48:dd308248-8ee8-49be-b928-07c3de7d7aa2.5145.5:8","obj_delete_at_hint.0000000124","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-23:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.23:8","obj_delete_at_hint.0000000001","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.4-tkscyf-4","obj_delete_at_hint.0000000126","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-16:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.16:3","obj_delete_at_hint.0000000055","obj_delete_at_hint.0000000061","datalog.sync-status.shard.92d097d8-38ae-4839-9777-3b30db2b2f60.2","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-20:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.20:8","obj_delete_at_hint.0000000097","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.20-tkscyf-20","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-21:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.21:8","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-15:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.15:1","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-18:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.18:3","datalog.sync-status.shard.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2.3","obj_delete_at_hint.0000000042","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-49:dd308248-8ee8-49be-b928-07c3de7d7aa2.5352.1:5","obj_delete_at_hint.0000000075","obj_delete_at_hint.0000000120","obj_delete_at_hint.0000000030","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.36-tkscyf-36"] 2021-02-11T10:01:01.517 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:01:01.518 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:01:01.518 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:01:01.552 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:01:01.552 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:01:01.823 INFO:teuthology.orchestra.run.smithi098.stdout:["obj_delete_at_hint.0000000078","meta.history","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-10:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.10:8","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.37-tkscyf-37","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-21:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.21:8","obj_delete_at_hint.0000000070","obj_delete_at_hint.0000000104","obj_delete_at_hint.0000000026","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.22-tkscyf-22","2021-02-11-09-63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2.5092.2-tkscyf-43","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-48:dd308248-8ee8-49be-b928-07c3de7d7aa2.5145.5:8","obj_delete_at_hint.0000000028","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-26:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.26:8","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-48:dd308248-8ee8-49be-b928-07c3de7d7aa2.5145.5:8","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.15-tkscyf-15","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-11:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.11:7","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.26-tkscyf-26","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.5352.1-tkscyf-49","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.17-tkscyf-17","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-37:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.37:10","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-17:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.17:5","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.40-tkscyf-40","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.6-tkscyf-6","obj_delete_at_hint.0000000040","meta.full-sync.index.0","obj_delete_at_hint.0000000015","obj_delete_at_hint.0000000069","obj_delete_at_hint.0000000095","obj_delete_at_hint.0000000003","obj_delete_at_hint.0000000047","obj_delete_at_hint.0000000035","obj_delete_at_hint.0000000037","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-15:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.15:1","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.31-tkscyf-31","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.5145.4-tkscyf-47","obj_delete_at_hint.0000000024","obj_delete_at_hint.0000000000","obj_delete_at_hint.0000000031","data_log.3","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-10:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.10:3","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.5145.3-tkscyf-46","data_loggenerations_metadata","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-20:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.20:8","2021-02-11-09-63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2.5092.1-tkscyf-42","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-23:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.23:8","obj_delete_at_hint.0000000076","obj_delete_at_hint.0000000113","obj_delete_at_hint.0000000005","obj_delete_at_hint.0000000011","obj_delete_at_hint.0000000058","obj_delete_at_hint.0000000052","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.19-tkscyf-19","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-25:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.25:8","data.full-sync.index.dd308248-8ee8-49be-b928-07c3de7d7aa2.1","data.full-sync.index.dd308248-8ee8-49be-b928-07c3de7d7aa2.0","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-11:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.11:8","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-11:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.11:1","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-11:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.11:8","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-18:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.18:5","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.5145.5-tkscyf-48","obj_delete_at_hint.0000000088","obj_delete_at_hint.0000000080","obj_delete_at_hint.0000000090","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.23-tkscyf-23","obj_delete_at_hint.0000000110","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.13-tkscyf-13","obj_delete_at_hint.0000000096","obj_delete_at_hint.0000000087","obj_delete_at_hint.0000000008","datalog.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-17:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.17:1","obj_delete_at_hint.0000000006","obj_delete_at_hint.0000000029","obj_delete_at_hint.0000000089","obj_delete_at_hint.0000000022","meta.log.0d37c290-12ce-40b6-8dd3-c3f1d036f1a7.0","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-49:dd308248-8ee8-49be-b928-07c3de7d7aa2.5352.1:5","bilog.trim","obj_delete_at_hint.0000000016","obj_delete_at_hint.0000000041","data.full-sync.index.92d097d8-38ae-4839-9777-3b30db2b2f60.0","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-10:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.10:3","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-12:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.12:8","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.11-tkscyf-11","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-19:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.19:5","obj_delete_at_hint.0000000018","obj_delete_at_hint.0000000092","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-16:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.16:5","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-11:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.11:7","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-17:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.17:5","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-16:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.16:3","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-10:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.10:7","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-12:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.12:7","obj_delete_at_hint.0000000014","obj_delete_at_hint.0000000112","obj_delete_at_hint.0000000007","obj_delete_at_hint.0000000021","obj_delete_at_hint.0000000064","obj_delete_at_hint.0000000071","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.5-tkscyf-5","obj_delete_at_hint.0000000074","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-19:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.19:8","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-12:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.12:3","obj_delete_at_hint.0000000081","obj_delete_at_hint.0000000009","data.full-sync.index.92d097d8-38ae-4839-9777-3b30db2b2f60.2","obj_delete_at_hint.0000000121","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.9-tkscyf-9","data_log.1","obj_delete_at_hint.0000000125","obj_delete_at_hint.0000000082","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-16:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.16:5","meta.full-sync.index.1","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-23:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.23:8","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-10:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.10:8","obj_delete_at_hint.0000000105","obj_delete_at_hint.0000000059","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.12-tkscyf-12","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.8-tkscyf-8","obj_delete_at_hint.0000000077","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-18:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.18:3","obj_delete_at_hint.0000000032","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-12:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.12:7","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-11:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.11:3","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-38:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.38:10","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-12:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.12:1","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.25-tkscyf-25","data.full-sync.index.dd308248-8ee8-49be-b928-07c3de7d7aa2.3","obj_delete_at_hint.0000000053","obj_delete_at_hint.0000000091","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.3-tkscyf-3","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.5145.1-tkscyf-44","obj_delete_at_hint.0000000065","obj_delete_at_hint.0000000083","obj_delete_at_hint.0000000010","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-16:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.16:1","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-18:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.18:5","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-36:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.36:10","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.41-tkscyf-41","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-14:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.14:1","mdlog.sync-status.shard.2","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-17:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.17:3","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.24-tkscyf-24","obj_delete_at_hint.0000000045","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-22:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.22:8","obj_delete_at_hint.0000000002","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-36:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.36:10","obj_delete_at_hint.0000000116","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-18:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.18:1","obj_delete_at_hint.0000000034","obj_delete_at_hint.0000000101","mdlog.sync-status.shard.1","datalog.sync-status.shard.dd308248-8ee8-49be-b928-07c3de7d7aa2.0","obj_delete_at_hint.0000000079","obj_delete_at_hint.0000000049","obj_delete_at_hint.0000000117","obj_delete_at_hint.0000000044","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-26:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.26:8","obj_delete_at_hint.0000000066","obj_delete_at_hint.0000000068","obj_delete_at_hint.0000000085","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.18-tkscyf-18","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-38:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.38:10","obj_delete_at_hint.0000000073","obj_delete_at_hint.0000000038","datalog.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60","meta.full-sync.index.3","obj_delete_at_hint.0000000118","obj_delete_at_hint.0000000036","obj_delete_at_hint.0000000103","mdlog.sync-status.shard.3","obj_delete_at_hint.0000000119","obj_delete_at_hint.0000000098","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.34-tkscyf-34","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.5145.2-tkscyf-45","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.38-tkscyf-38","mdlog.sync-status.shard.0","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-10:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.10:1","obj_delete_at_hint.0000000027","data.full-sync.index.92d097d8-38ae-4839-9777-3b30db2b2f60.3","mdlog.sync-status","obj_delete_at_hint.0000000019","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-13:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.13:1","obj_delete_at_hint.0000000039","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-12:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.12:3","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-46:dd308248-8ee8-49be-b928-07c3de7d7aa2.5145.3:8","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-12:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.12:8","obj_delete_at_hint.0000000100","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.14-tkscyf-14","obj_delete_at_hint.0000000093","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-22:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.22:8","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-17:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.17:3","obj_delete_at_hint.0000000004","obj_delete_at_hint.0000000063","obj_delete_at_hint.0000000122","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-25:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.25:8","obj_delete_at_hint.0000000057","obj_delete_at_hint.0000000054","obj_delete_at_hint.0000000114","data_log.0","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-24:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.24:8","obj_delete_at_hint.0000000012","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-46:dd308248-8ee8-49be-b928-07c3de7d7aa2.5145.3:8","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.10-tkscyf-10","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.1-tkscyf-1","obj_delete_at_hint.0000000084","obj_delete_at_hint.0000000043","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-18:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.18:1","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-17:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.17:1","obj_delete_at_hint.0000000111","datalog.sync-status.shard.92d097d8-38ae-4839-9777-3b30db2b2f60.1","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-19:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.19:8","obj_delete_at_hint.0000000048","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.2-tkscyf-51","obj_delete_at_hint.0000000020","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-10:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.10:1","obj_delete_at_hint.0000000099","obj_delete_at_hint.0000000056","data.full-sync.index.92d097d8-38ae-4839-9777-3b30db2b2f60.1","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-10:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.10:7","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-13:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.13:1","obj_delete_at_hint.0000000072","obj_delete_at_hint.0000000062","obj_delete_at_hint.0000000115","obj_delete_at_hint.0000000033","data.full-sync.index.dd308248-8ee8-49be-b928-07c3de7d7aa2.2","obj_delete_at_hint.0000000025","obj_delete_at_hint.0000000123","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.21-tkscyf-21","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-19:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.19:5","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-47:dd308248-8ee8-49be-b928-07c3de7d7aa2.5145.4:8","datalog.sync-status.shard.dd308248-8ee8-49be-b928-07c3de7d7aa2.2","obj_delete_at_hint.0000000108","obj_delete_at_hint.0000000094","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.16-tkscyf-16","obj_delete_at_hint.0000000060","obj_delete_at_hint.0000000109","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-14:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.14:1","obj_delete_at_hint.0000000013","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-16:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.16:1","obj_delete_at_hint.0000000051","obj_delete_at_hint.0000000106","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.39-tkscyf-39","obj_delete_at_hint.0000000017","obj_delete_at_hint.0000000023","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.5-tkscyf-54","datalog.sync-status.shard.dd308248-8ee8-49be-b928-07c3de7d7aa2.3","obj_delete_at_hint.0000000107","obj_delete_at_hint.0000000046","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-37:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.37:10","meta.full-sync.index.2","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-24:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.24:8","data_log.2","obj_delete_at_hint.0000000067","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-18:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.18:3","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.8-tkscyf-57","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.2-tkscyf-2","obj_delete_at_hint.0000000086","datalog.sync-status.shard.92d097d8-38ae-4839-9777-3b30db2b2f60.0","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-12:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.12:1","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.7-tkscyf-7","datalog.sync-status.shard.92d097d8-38ae-4839-9777-3b30db2b2f60.3","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-47:dd308248-8ee8-49be-b928-07c3de7d7aa2.5145.4:8","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-11:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.11:3","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.28-tkscyf-28","obj_delete_at_hint.0000000050","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-11:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.11:1","obj_delete_at_hint.0000000102","obj_delete_at_hint.0000000124","datalog.sync-status.shard.dd308248-8ee8-49be-b928-07c3de7d7aa2.1","obj_delete_at_hint.0000000001","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.4-tkscyf-4","obj_delete_at_hint.0000000126","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-16:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.16:3","obj_delete_at_hint.0000000055","obj_delete_at_hint.0000000061","datalog.sync-status.shard.92d097d8-38ae-4839-9777-3b30db2b2f60.2","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-20:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.20:8","obj_delete_at_hint.0000000097","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.20-tkscyf-20","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-21:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.21:8","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-15:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.15:1","obj_delete_at_hint.0000000042","bucket.sync-status.92d097d8-38ae-4839-9777-3b30db2b2f60:tkscyf-49:dd308248-8ee8-49be-b928-07c3de7d7aa2.5352.1:5","obj_delete_at_hint.0000000075","obj_delete_at_hint.0000000120","obj_delete_at_hint.0000000030","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.36-tkscyf-36"] 2021-02-11T10:01:01.858 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:01:01.858 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:01:01.859 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:01:01.897 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:01:01.897 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:01:02.126 INFO:teuthology.orchestra.run.smithi094.stdout:["obj_delete_at_hint.0000000078","meta.history","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-16:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.16:5","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.37-tkscyf-37","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-21:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.21:8","obj_delete_at_hint.0000000070","obj_delete_at_hint.0000000104","obj_delete_at_hint.0000000026","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.22-tkscyf-22","2021-02-11-09-63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2.5092.2-tkscyf-43","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-26:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.26:8","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-48:dd308248-8ee8-49be-b928-07c3de7d7aa2.5145.5:8","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-11:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.11:7","obj_delete_at_hint.0000000028","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.15-tkscyf-15","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.26-tkscyf-26","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.5352.1-tkscyf-49","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-12:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.12:7","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.17-tkscyf-17","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-17:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.17:5","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.40-tkscyf-40","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.6-tkscyf-6","obj_delete_at_hint.0000000040","meta.full-sync.index.0","obj_delete_at_hint.0000000015","obj_delete_at_hint.0000000069","obj_delete_at_hint.0000000095","obj_delete_at_hint.0000000003","obj_delete_at_hint.0000000047","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-10:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.10:8","obj_delete_at_hint.0000000035","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-24:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.24:8","obj_delete_at_hint.0000000037","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-15:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.15:1","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-16:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.16:1","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.5145.4-tkscyf-47","obj_delete_at_hint.0000000024","obj_delete_at_hint.0000000000","obj_delete_at_hint.0000000031","data_log.3","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.5145.3-tkscyf-46","data_loggenerations_metadata","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-20:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.20:8","2021-02-11-09-63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2.5092.1-tkscyf-42","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-49:dd308248-8ee8-49be-b928-07c3de7d7aa2.5352.1:5","obj_delete_at_hint.0000000076","obj_delete_at_hint.0000000113","obj_delete_at_hint.0000000005","obj_delete_at_hint.0000000011","obj_delete_at_hint.0000000058","obj_delete_at_hint.0000000052","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.19-tkscyf-19","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-25:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.25:8","data.full-sync.index.dd308248-8ee8-49be-b928-07c3de7d7aa2.1","data.full-sync.index.dd308248-8ee8-49be-b928-07c3de7d7aa2.0","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.29-tkscyf-29","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-12:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.12:3","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-11:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.11:1","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-11:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.11:8","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-25:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.25:8","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.5145.5-tkscyf-48","obj_delete_at_hint.0000000088","obj_delete_at_hint.0000000080","obj_delete_at_hint.0000000090","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.23-tkscyf-23","obj_delete_at_hint.0000000110","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.13-tkscyf-13","obj_delete_at_hint.0000000096","obj_delete_at_hint.0000000087","obj_delete_at_hint.0000000008","datalog.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-17:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.17:1","obj_delete_at_hint.0000000006","obj_delete_at_hint.0000000029","obj_delete_at_hint.0000000089","obj_delete_at_hint.0000000022","meta.log.0d37c290-12ce-40b6-8dd3-c3f1d036f1a7.0","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-49:dd308248-8ee8-49be-b928-07c3de7d7aa2.5352.1:5","bilog.trim","obj_delete_at_hint.0000000016","obj_delete_at_hint.0000000041","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-10:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.10:3","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.11-tkscyf-11","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-19:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.19:5","obj_delete_at_hint.0000000018","obj_delete_at_hint.0000000092","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-10:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.10:7","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-16:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.16:5","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-11:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.11:7","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-16:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.16:3","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-12:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.12:7","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-13:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.13:1","obj_delete_at_hint.0000000014","obj_delete_at_hint.0000000112","obj_delete_at_hint.0000000007","obj_delete_at_hint.0000000021","obj_delete_at_hint.0000000064","obj_delete_at_hint.0000000071","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.5-tkscyf-5","obj_delete_at_hint.0000000074","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-18:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.18:5","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-19:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.19:8","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-17:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.17:5","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.32-tkscyf-32","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-12:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.12:3","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.3-tkscyf-52","obj_delete_at_hint.0000000081","obj_delete_at_hint.0000000009","obj_delete_at_hint.0000000121","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.9-tkscyf-9","data_log.1","obj_delete_at_hint.0000000125","obj_delete_at_hint.0000000082","meta.full-sync.index.1","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-37:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.37:10","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-23:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.23:8","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-10:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.10:8","obj_delete_at_hint.0000000105","obj_delete_at_hint.0000000059","datalog.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.12-tkscyf-12","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.8-tkscyf-8","obj_delete_at_hint.0000000077","obj_delete_at_hint.0000000032","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-11:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.11:3","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.9-tkscyf-58","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.25-tkscyf-25","data.full-sync.index.dd308248-8ee8-49be-b928-07c3de7d7aa2.3","obj_delete_at_hint.0000000053","obj_delete_at_hint.0000000091","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.3-tkscyf-3","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.5145.1-tkscyf-44","obj_delete_at_hint.0000000065","obj_delete_at_hint.0000000083","obj_delete_at_hint.0000000010","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-20:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.20:8","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-16:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.16:1","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-10:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.10:3","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-18:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.18:5","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.41-tkscyf-41","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-11:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.11:1","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-14:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.14:1","mdlog.sync-status.shard.2","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-17:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.17:3","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.24-tkscyf-24","data.full-sync.index.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2.1","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.35-tkscyf-35","obj_delete_at_hint.0000000045","obj_delete_at_hint.0000000002","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-36:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.36:10","obj_delete_at_hint.0000000116","obj_delete_at_hint.0000000034","obj_delete_at_hint.0000000101","mdlog.sync-status.shard.1","meta.log.47caa906-66a7-4be6-bbb9-9f539accb732.0","datalog.sync-status.shard.dd308248-8ee8-49be-b928-07c3de7d7aa2.0","obj_delete_at_hint.0000000079","obj_delete_at_hint.0000000049","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-16:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.16:3","obj_delete_at_hint.0000000117","obj_delete_at_hint.0000000044","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-38:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.38:10","meta.log.ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1.1","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-26:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.26:8","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-12:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.12:8","obj_delete_at_hint.0000000066","obj_delete_at_hint.0000000068","obj_delete_at_hint.0000000085","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.18-tkscyf-18","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-38:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.38:10","obj_delete_at_hint.0000000073","obj_delete_at_hint.0000000038","meta.full-sync.index.3","obj_delete_at_hint.0000000118","obj_delete_at_hint.0000000036","obj_delete_at_hint.0000000103","mdlog.sync-status.shard.3","obj_delete_at_hint.0000000119","obj_delete_at_hint.0000000098","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.5145.2-tkscyf-45","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-18:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.18:1","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.38-tkscyf-38","mdlog.sync-status.shard.0","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-10:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.10:1","obj_delete_at_hint.0000000027","mdlog.sync-status","obj_delete_at_hint.0000000019","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.6-tkscyf-55","obj_delete_at_hint.0000000039","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-46:dd308248-8ee8-49be-b928-07c3de7d7aa2.5145.3:8","datalog.sync-status.shard.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2.1","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-12:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.12:8","obj_delete_at_hint.0000000100","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.14-tkscyf-14","obj_delete_at_hint.0000000093","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-22:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.22:8","obj_delete_at_hint.0000000004","obj_delete_at_hint.0000000063","obj_delete_at_hint.0000000122","obj_delete_at_hint.0000000057","datalog.sync-status.shard.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2.2","obj_delete_at_hint.0000000054","obj_delete_at_hint.0000000114","data_log.0","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-24:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.24:8","obj_delete_at_hint.0000000012","data.full-sync.index.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2.0","meta.log.ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1.0","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.10-tkscyf-10","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.1-tkscyf-1","obj_delete_at_hint.0000000084","obj_delete_at_hint.0000000043","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-18:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.18:1","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-17:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.17:3","obj_delete_at_hint.0000000111","obj_delete_at_hint.0000000048","obj_delete_at_hint.0000000020","datalog.sync-status.shard.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2.0","data.full-sync.index.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2.2","obj_delete_at_hint.0000000099","obj_delete_at_hint.0000000056","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-10:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.10:1","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-10:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.10:7","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-13:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.13:1","obj_delete_at_hint.0000000072","obj_delete_at_hint.0000000062","obj_delete_at_hint.0000000115","obj_delete_at_hint.0000000033","data.full-sync.index.dd308248-8ee8-49be-b928-07c3de7d7aa2.2","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-15:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.15:1","data.full-sync.index.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2.3","obj_delete_at_hint.0000000025","obj_delete_at_hint.0000000123","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.21-tkscyf-21","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-47:dd308248-8ee8-49be-b928-07c3de7d7aa2.5145.4:8","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-14:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.14:1","datalog.sync-status.shard.dd308248-8ee8-49be-b928-07c3de7d7aa2.2","obj_delete_at_hint.0000000108","obj_delete_at_hint.0000000094","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.16-tkscyf-16","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-47:dd308248-8ee8-49be-b928-07c3de7d7aa2.5145.4:8","obj_delete_at_hint.0000000060","obj_delete_at_hint.0000000109","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-46:dd308248-8ee8-49be-b928-07c3de7d7aa2.5145.3:8","obj_delete_at_hint.0000000013","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-19:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.19:5","obj_delete_at_hint.0000000051","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-36:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.36:10","obj_delete_at_hint.0000000106","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.39-tkscyf-39","obj_delete_at_hint.0000000017","obj_delete_at_hint.0000000023","datalog.sync-status.shard.dd308248-8ee8-49be-b928-07c3de7d7aa2.3","obj_delete_at_hint.0000000107","obj_delete_at_hint.0000000046","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-37:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.37:10","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-11:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.11:8","meta.full-sync.index.2","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-19:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.19:8","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-11:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.11:3","data_log.2","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-17:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.17:1","obj_delete_at_hint.0000000067","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-18:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.18:3","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.2-tkscyf-2","obj_delete_at_hint.0000000086","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-12:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.12:1","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.7-tkscyf-7","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-22:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.22:8","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-12:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.12:1","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-21:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.21:8","obj_delete_at_hint.0000000050","obj_delete_at_hint.0000000102","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-48:dd308248-8ee8-49be-b928-07c3de7d7aa2.5145.5:8","obj_delete_at_hint.0000000124","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-23:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.23:8","datalog.sync-status.shard.dd308248-8ee8-49be-b928-07c3de7d7aa2.1","obj_delete_at_hint.0000000001","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.4-tkscyf-4","obj_delete_at_hint.0000000126","obj_delete_at_hint.0000000055","obj_delete_at_hint.0000000061","obj_delete_at_hint.0000000097","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.20-tkscyf-20","bucket.sync-status.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2:tkscyf-18:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.18:3","datalog.sync-status.shard.63e9bff1-2a4a-47fa-9ee6-d55fa18cc0c2.3","obj_delete_at_hint.0000000042","obj_delete_at_hint.0000000075","obj_delete_at_hint.0000000120","obj_delete_at_hint.0000000030","2021-02-11-09-dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.36-tkscyf-36"] 2021-02-11T10:01:02.155 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:01:02.155 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:01:02.155 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone4 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:01:02.196 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:01:02.196 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:01:02.414 INFO:teuthology.orchestra.run.smithi098.stdout:["obj_delete_at_hint.0000000078","meta.history","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-21:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.21:8","obj_delete_at_hint.0000000070","obj_delete_at_hint.0000000104","obj_delete_at_hint.0000000026","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-48:dd308248-8ee8-49be-b928-07c3de7d7aa2.5145.5:8","obj_delete_at_hint.0000000028","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-17:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.17:5","obj_delete_at_hint.0000000040","meta.full-sync.index.0","obj_delete_at_hint.0000000015","obj_delete_at_hint.0000000069","obj_delete_at_hint.0000000095","obj_delete_at_hint.0000000003","obj_delete_at_hint.0000000047","obj_delete_at_hint.0000000035","obj_delete_at_hint.0000000037","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-15:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.15:1","obj_delete_at_hint.0000000024","obj_delete_at_hint.0000000000","obj_delete_at_hint.0000000031","data_log.3","data_loggenerations_metadata","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-20:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.20:8","obj_delete_at_hint.0000000076","obj_delete_at_hint.0000000113","obj_delete_at_hint.0000000005","obj_delete_at_hint.0000000011","obj_delete_at_hint.0000000058","obj_delete_at_hint.0000000052","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-25:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.25:8","data.full-sync.index.dd308248-8ee8-49be-b928-07c3de7d7aa2.1","data.full-sync.index.dd308248-8ee8-49be-b928-07c3de7d7aa2.0","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-11:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.11:1","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-11:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.11:8","obj_delete_at_hint.0000000088","obj_delete_at_hint.0000000080","obj_delete_at_hint.0000000090","obj_delete_at_hint.0000000110","obj_delete_at_hint.0000000096","obj_delete_at_hint.0000000087","obj_delete_at_hint.0000000008","datalog.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-17:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.17:1","obj_delete_at_hint.0000000006","obj_delete_at_hint.0000000029","obj_delete_at_hint.0000000089","obj_delete_at_hint.0000000022","meta.log.0d37c290-12ce-40b6-8dd3-c3f1d036f1a7.0","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-49:dd308248-8ee8-49be-b928-07c3de7d7aa2.5352.1:5","bilog.trim","obj_delete_at_hint.0000000016","obj_delete_at_hint.0000000041","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-10:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.10:3","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-19:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.19:5","obj_delete_at_hint.0000000018","obj_delete_at_hint.0000000092","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-16:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.16:5","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-11:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.11:7","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-16:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.16:3","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-12:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.12:7","obj_delete_at_hint.0000000014","obj_delete_at_hint.0000000112","obj_delete_at_hint.0000000007","obj_delete_at_hint.0000000021","obj_delete_at_hint.0000000064","obj_delete_at_hint.0000000071","obj_delete_at_hint.0000000074","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-19:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.19:8","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-12:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.12:3","obj_delete_at_hint.0000000081","obj_delete_at_hint.0000000009","obj_delete_at_hint.0000000121","data_log.1","obj_delete_at_hint.0000000125","obj_delete_at_hint.0000000082","meta.full-sync.index.1","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-23:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.23:8","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-10:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.10:8","obj_delete_at_hint.0000000105","obj_delete_at_hint.0000000059","obj_delete_at_hint.0000000077","obj_delete_at_hint.0000000032","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-11:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.11:3","data.full-sync.index.dd308248-8ee8-49be-b928-07c3de7d7aa2.3","obj_delete_at_hint.0000000053","obj_delete_at_hint.0000000091","obj_delete_at_hint.0000000065","obj_delete_at_hint.0000000083","obj_delete_at_hint.0000000010","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-16:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.16:1","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-18:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.18:5","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-14:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.14:1","mdlog.sync-status.shard.2","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-17:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.17:3","obj_delete_at_hint.0000000045","obj_delete_at_hint.0000000002","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-36:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.36:10","obj_delete_at_hint.0000000116","obj_delete_at_hint.0000000034","obj_delete_at_hint.0000000101","mdlog.sync-status.shard.1","meta.log.47caa906-66a7-4be6-bbb9-9f539accb732.0","datalog.sync-status.shard.dd308248-8ee8-49be-b928-07c3de7d7aa2.0","obj_delete_at_hint.0000000079","obj_delete_at_hint.0000000049","obj_delete_at_hint.0000000117","obj_delete_at_hint.0000000044","meta.log.ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1.1","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-26:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.26:8","obj_delete_at_hint.0000000066","obj_delete_at_hint.0000000068","obj_delete_at_hint.0000000085","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-38:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.38:10","obj_delete_at_hint.0000000073","obj_delete_at_hint.0000000038","meta.full-sync.index.3","obj_delete_at_hint.0000000118","obj_delete_at_hint.0000000036","obj_delete_at_hint.0000000103","mdlog.sync-status.shard.3","obj_delete_at_hint.0000000119","obj_delete_at_hint.0000000098","mdlog.sync-status.shard.0","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-10:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.10:1","obj_delete_at_hint.0000000027","mdlog.sync-status","obj_delete_at_hint.0000000019","obj_delete_at_hint.0000000039","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-46:dd308248-8ee8-49be-b928-07c3de7d7aa2.5145.3:8","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-12:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.12:8","obj_delete_at_hint.0000000100","obj_delete_at_hint.0000000093","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-22:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.22:8","obj_delete_at_hint.0000000004","obj_delete_at_hint.0000000063","obj_delete_at_hint.0000000122","obj_delete_at_hint.0000000057","obj_delete_at_hint.0000000054","obj_delete_at_hint.0000000114","data_log.0","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-24:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.24:8","obj_delete_at_hint.0000000012","meta.log.ccd64ac3-c848-4ba5-a8d4-fa08a8d1e6f1.0","obj_delete_at_hint.0000000084","obj_delete_at_hint.0000000043","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-18:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.18:1","obj_delete_at_hint.0000000111","obj_delete_at_hint.0000000048","obj_delete_at_hint.0000000020","obj_delete_at_hint.0000000099","obj_delete_at_hint.0000000056","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-10:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.10:7","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-13:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.13:1","obj_delete_at_hint.0000000072","obj_delete_at_hint.0000000062","obj_delete_at_hint.0000000115","obj_delete_at_hint.0000000033","data.full-sync.index.dd308248-8ee8-49be-b928-07c3de7d7aa2.2","obj_delete_at_hint.0000000025","obj_delete_at_hint.0000000123","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-47:dd308248-8ee8-49be-b928-07c3de7d7aa2.5145.4:8","datalog.sync-status.shard.dd308248-8ee8-49be-b928-07c3de7d7aa2.2","obj_delete_at_hint.0000000108","obj_delete_at_hint.0000000094","obj_delete_at_hint.0000000060","obj_delete_at_hint.0000000109","obj_delete_at_hint.0000000013","obj_delete_at_hint.0000000051","obj_delete_at_hint.0000000106","obj_delete_at_hint.0000000017","obj_delete_at_hint.0000000023","datalog.sync-status.shard.dd308248-8ee8-49be-b928-07c3de7d7aa2.3","obj_delete_at_hint.0000000107","obj_delete_at_hint.0000000046","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-37:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.37:10","meta.full-sync.index.2","data_log.2","obj_delete_at_hint.0000000067","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-18:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.18:3","obj_delete_at_hint.0000000086","bucket.sync-status.dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-12:dd308248-8ee8-49be-b928-07c3de7d7aa2.4339.12:1","obj_delete_at_hint.0000000050","obj_delete_at_hint.0000000102","obj_delete_at_hint.0000000124","datalog.sync-status.shard.dd308248-8ee8-49be-b928-07c3de7d7aa2.1","obj_delete_at_hint.0000000001","obj_delete_at_hint.0000000126","obj_delete_at_hint.0000000055","obj_delete_at_hint.0000000061","obj_delete_at_hint.0000000097","obj_delete_at_hint.0000000042","obj_delete_at_hint.0000000075","obj_delete_at_hint.0000000120","obj_delete_at_hint.0000000030"] 2021-02-11T10:01:02.449 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:01:02.449 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:01:02.449 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:01:02.488 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:01:02.488 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:01:02.734 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613037661.149966_525.3","last_update":"2021-02-11T10:01:01.149966Z"},{"marker":"1_1613037661.149966_363.3","last_update":"2021-02-11T10:01:01.149966Z"},{"marker":"1_1613037661.149966_205.2","last_update":"2021-02-11T10:01:01.149966Z"},{"marker":"1_1613037661.149966_272.3","last_update":"2021-02-11T10:01:01.149966Z"}] 2021-02-11T10:01:02.734 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-02-11T10:01:02.734 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:01:02.734 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:01:02.734 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:01:02.774 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:01:02.774 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:01:03.033 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4703921989677143155},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037401.531874_444.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:56:41.531874Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037058.105459_155.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:50:58.105459Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613036942.835890_67.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:49:02.835890Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037207.348502_176.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:53:27.348502Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:01:03.034 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1613037661.149966_525.3 target=1_1613037401.531874_444.1, shard=1 master=1_1613037661.149966_363.3 target=1_1613037058.105459_155.1, shard=2 master=1_1613037661.149966_205.2 target=1_1613036942.835890_67.3, shard=3 master=1_1613037661.149966_272.3 target=1_1613037207.348502_176.1 2021-02-11T10:01:08.035 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:01:08.035 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:01:08.035 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:01:08.080 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:01:08.080 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:01:08.331 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4703921989677143155},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037401.531874_444.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:56:41.531874Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037058.105459_155.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:50:58.105459Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613036942.835890_67.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:49:02.835890Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037207.348502_176.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:53:27.348502Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:01:08.332 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1613037661.149966_525.3 target=1_1613037401.531874_444.1, shard=1 master=1_1613037661.149966_363.3 target=1_1613037058.105459_155.1, shard=2 master=1_1613037661.149966_205.2 target=1_1613036942.835890_67.3, shard=3 master=1_1613037661.149966_272.3 target=1_1613037207.348502_176.1 2021-02-11T10:01:13.333 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:01:13.333 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:01:13.333 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:01:13.376 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:01:13.377 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:01:13.628 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4703921989677143155},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037401.531874_444.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:56:41.531874Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037058.105459_155.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:50:58.105459Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613036942.835890_67.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:49:02.835890Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037207.348502_176.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:53:27.348502Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:01:13.629 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1613037661.149966_525.3 target=1_1613037401.531874_444.1, shard=1 master=1_1613037661.149966_363.3 target=1_1613037058.105459_155.1, shard=2 master=1_1613037661.149966_205.2 target=1_1613036942.835890_67.3, shard=3 master=1_1613037661.149966_272.3 target=1_1613037207.348502_176.1 2021-02-11T10:01:18.629 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:01:18.629 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:01:18.630 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:01:18.674 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:01:18.674 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:01:18.937 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4703921989677143155},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037661.149966_525.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:01:01.149966Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037661.149966_363.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:01:01.149966Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613037661.149966_205.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:01:01.149966Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037661.149966_272.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:01:01.149966Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:01:18.937 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-02-11T10:01:18.938 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:01:18.938 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:01:18.938 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:01:18.976 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:01:18.976 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:01:19.217 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613037661.149966_525.3","last_update":"2021-02-11T10:01:01.149966Z"},{"marker":"1_1613037661.149966_363.3","last_update":"2021-02-11T10:01:01.149966Z"},{"marker":"1_1613037661.149966_205.2","last_update":"2021-02-11T10:01:01.149966Z"},{"marker":"1_1613037661.149966_272.3","last_update":"2021-02-11T10:01:01.149966Z"}] 2021-02-11T10:01:19.218 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-02-11T10:01:19.218 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:01:19.218 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:01:19.218 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:01:19.306 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:01:19.307 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:01:19.538 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":13312279424886386302},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037661.149966_525.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:01:01.149966Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037661.149966_363.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:01:01.149966Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613037661.149966_205.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:01:01.149966Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037661.149966_272.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:01:01.149966Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:01:19.539 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-02-11T10:01:19.539 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:01:19.539 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:01:19.539 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:01:19.577 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:01:19.577 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:01:19.818 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613037661.149966_525.3","last_update":"2021-02-11T10:01:01.149966Z"},{"marker":"1_1613037661.149966_363.3","last_update":"2021-02-11T10:01:01.149966Z"},{"marker":"1_1613037661.149966_205.2","last_update":"2021-02-11T10:01:01.149966Z"},{"marker":"1_1613037661.149966_272.3","last_update":"2021-02-11T10:01:01.149966Z"}] 2021-02-11T10:01:19.818 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-02-11T10:01:19.819 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:01:19.819 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:01:19.819 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:01:19.862 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:01:19.863 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:01:20.106 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2896161496309628191},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037661.149966_525.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:01:01.149966Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037661.149966_363.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:01:01.149966Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613037661.149966_205.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:01:01.149966Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037661.149966_272.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:01:01.149966Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:01:20.107 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-02-11T10:01:20.107 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:01:20.107 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:01:20.107 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:01:20.188 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:01:20.188 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:01:20.446 INFO:teuthology.orchestra.run.smithi098.stdout:[{"marker":"1_1613037662.670057_236.1","last_update":"2021-02-11T10:01:02.670057Z"},{"marker":"1_1613037662.672369_385.1","last_update":"2021-02-11T10:01:02.672369Z"},{"marker":"1_1613037662.664954_239.1","last_update":"2021-02-11T10:01:02.664954Z"},{"marker":"1_1613037662.667245_304.1","last_update":"2021-02-11T10:01:02.667245Z"}] 2021-02-11T10:01:20.446 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-02-11T10:01:20.447 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:01:20.447 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:01:20.447 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:01:20.490 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:01:20.490 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:01:20.735 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":15226599365335882107},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037662.670057_236.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:01:02.670057Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037662.672369_385.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:01:02.672369Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613037662.664954_239.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:01:02.664954Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037662.667245_304.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:01:02.667245Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:01:20.735 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-02-11T10:01:20.735 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:01:20.735 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:01:20.735 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:01:20.775 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:01:20.775 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:01:21.009 INFO:teuthology.orchestra.run.smithi098.stdout:[{"marker":"1_1613037662.670057_236.1","last_update":"2021-02-11T10:01:02.670057Z"},{"marker":"1_1613037662.672369_385.1","last_update":"2021-02-11T10:01:02.672369Z"},{"marker":"1_1613037662.664954_239.1","last_update":"2021-02-11T10:01:02.664954Z"},{"marker":"1_1613037662.667245_304.1","last_update":"2021-02-11T10:01:02.667245Z"}] 2021-02-11T10:01:21.009 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-02-11T10:01:21.009 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:01:21.009 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:01:21.010 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:01:21.055 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:01:21.055 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:01:21.284 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":10999913493642558030},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037662.670057_236.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:01:02.670057Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037662.672369_385.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:01:02.672369Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613037662.664954_239.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:01:02.664954Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037662.667245_304.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:01:02.667245Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:01:21.284 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-02-11T10:01:21.284 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:01:21.284 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:01:21.284 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:01:21.320 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:01:21.321 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:01:21.550 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613037662.065493_287.1","last_update":"2021-02-11T10:01:02.065493Z"},{"marker":"1_1613037662.067349_287.1","last_update":"2021-02-11T10:01:02.067349Z"},{"marker":"1_1613037662.061883_160.1","last_update":"2021-02-11T10:01:02.061883Z"},{"marker":"1_1613037662.063781_211.1","last_update":"2021-02-11T10:01:02.063781Z"}] 2021-02-11T10:01:21.551 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-02-11T10:01:21.551 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:01:21.551 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:01:21.551 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:01:21.636 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:01:21.636 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:01:21.887 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2753764877348348355},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037662.065493_287.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:01:02.065493Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037662.067349_287.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:01:02.067349Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613037662.061883_160.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:01:02.061883Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037662.063781_211.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:01:02.063781Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:01:21.887 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-02-11T10:01:21.887 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:01:21.887 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:01:21.888 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:01:21.968 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:01:21.968 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:01:22.203 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613037662.065493_287.1","last_update":"2021-02-11T10:01:02.065493Z"},{"marker":"1_1613037662.067349_287.1","last_update":"2021-02-11T10:01:02.067349Z"},{"marker":"1_1613037662.061883_160.1","last_update":"2021-02-11T10:01:02.061883Z"},{"marker":"1_1613037662.063781_211.1","last_update":"2021-02-11T10:01:02.063781Z"}] 2021-02-11T10:01:22.203 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-02-11T10:01:22.203 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:01:22.204 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:01:22.204 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:01:22.246 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:01:22.246 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:01:22.497 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":7143279782539285748},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037662.065493_287.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:01:02.065493Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037662.067349_287.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:01:02.067349Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613037662.061883_160.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:01:02.061883Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037662.063781_211.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:01:02.063781Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:01:22.497 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-02-11T10:01:22.498 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_sync_disable ... ok 2021-02-11T10:01:22.498 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=tkscyf-59 2021-02-11T10:01:22.544 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=tkscyf-60 2021-02-11T10:01:22.622 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=tkscyf-61 2021-02-11T10:01:23.279 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:01:23.279 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:01:23.279 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:01:23.323 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:01:23.323 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:01:23.490 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 2021-02-11T10:01:23.517 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613037682.667764_499.1","last_update":"2021-02-11T10:01:22.667764Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:01:23.518 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T10:01:23.518 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:01:23.518 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:01:23.518 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:01:23.557 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:01:23.558 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:01:23.807 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037682.667764_499.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:01:22.667764Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:01:23.807 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T10:01:23.807 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T10:01:23.807 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:01:23.808 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:01:23.808 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:01:23.850 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:01:23.850 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:01:24.092 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037682.667764_499.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:01:22.667764Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:01:24.093 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T10:01:24.093 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:01:24.093 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:01:24.093 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:01:24.093 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:01:24.132 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:01:24.132 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:01:24.375 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037682.667764_499.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:01:22.667764Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:01:24.375 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:01:24.375 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:01:24.375 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:01:24.375 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-59 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:01:24.465 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:01:24.465 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:01:24.713 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:01:24.706+0000 7fdc60248080 1 bucket sync caught up with source: 2021-02-11T10:01:24.713 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [00000000001.63.5, , 00000000001.41.5, , 00000000001.53.5, , , 00000000001.43.5, , , ] 2021-02-11T10:01:24.713 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [00000000001.63.5, , 00000000001.41.5, , 00000000001.53.5, , , 00000000001.43.5, , , ] 2021-02-11T10:01:24.714 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:01:24.706+0000 7fdc60248080 0 bucket checkpoint complete 2021-02-11T10:01:24.749 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:01:24.749 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:01:24.749 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-59 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:01:24.796 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:01:24.796 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:01:25.012 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:01:25.007+0000 7f47116e3080 1 bucket sync caught up with source: 2021-02-11T10:01:25.013 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [00000000001.63.5, , 00000000001.41.5, , 00000000001.53.5, , , 00000000001.43.5, , , ] 2021-02-11T10:01:25.013 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [00000000001.63.5, , 00000000001.41.5, , 00000000001.53.5, , , 00000000001.43.5, , , ] 2021-02-11T10:01:25.013 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:01:25.007+0000 7f47116e3080 0 bucket checkpoint complete 2021-02-11T10:01:25.043 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:01:25.043 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:01:25.043 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-59 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:01:25.091 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:01:25.092 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:01:25.308 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:01:25.302+0000 7f260c2cc080 1 bucket sync caught up with source: 2021-02-11T10:01:25.309 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [00000000001.63.5, , 00000000001.41.5, , 00000000001.53.5, , , 00000000001.43.5, , , ] 2021-02-11T10:01:25.309 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [00000000001.63.5, , 00000000001.41.5, , 00000000001.53.5, , , 00000000001.43.5, , , ] 2021-02-11T10:01:25.309 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:01:25.302+0000 7f260c2cc080 0 bucket checkpoint complete 2021-02-11T10:01:25.343 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-59', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:01:25.343 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-59', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:01:25.344 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-59 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:01:25.388 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:01:25.388 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:01:25.613 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:01:25.607+0000 7f46c1609080 1 waiting for incremental sync to catch up: 2021-02-11T10:01:25.614 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , 00000000001.60.5, , 00000000001.56.5, , , 00000000001.64.5, , , ] 2021-02-11T10:01:25.614 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [00000000001.49.5, , 00000000001.60.5, , 00000000001.56.5, , , 00000000001.64.5, , , ] 2021-02-11T10:01:30.618 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:01:30.611+0000 7f46c1609080 1 waiting for incremental sync to catch up: 2021-02-11T10:01:30.618 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , 00000000001.60.5, , 00000000001.56.5, , , 00000000001.64.5, , , ] 2021-02-11T10:01:30.618 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [00000000001.49.5, , 00000000001.60.5, , 00000000001.56.5, , , 00000000001.64.5, , , ] 2021-02-11T10:01:35.622 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:01:35.615+0000 7f46c1609080 1 bucket sync caught up with source: 2021-02-11T10:01:44.299 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [00000000001.49.5, , 00000000001.60.5, , 00000000001.56.5, , , 00000000001.64.5, , , ] 2021-02-11T10:01:44.299 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [00000000001.49.5, , 00000000001.60.5, , 00000000001.56.5, , , 00000000001.64.5, , , ] 2021-02-11T10:01:44.299 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:01:35.615+0000 7f46c1609080 0 bucket checkpoint complete 2021-02-11T10:01:44.301 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-59', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:01:44.301 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-59', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:01:44.301 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-59 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:01:44.349 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:01:44.349 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:01:44.558 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:01:44.551+0000 7efe9f234080 1 bucket sync caught up with source: 2021-02-11T10:01:44.558 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [00000000001.49.5, , 00000000001.60.5, , 00000000001.56.5, , , 00000000001.64.5, , , ] 2021-02-11T10:01:44.558 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [00000000001.49.5, , 00000000001.60.5, , 00000000001.56.5, , , 00000000001.64.5, , , ] 2021-02-11T10:01:44.558 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:01:44.551+0000 7efe9f234080 0 bucket checkpoint complete 2021-02-11T10:01:44.597 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-59', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:01:44.597 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-59', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:01:44.598 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-59 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:01:44.640 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:01:44.641 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:01:44.866 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:01:44.859+0000 7f9fb8d3a080 1 bucket sync caught up with source: 2021-02-11T10:01:44.866 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [00000000001.55.5, , 00000000001.44.5, , 00000000001.59.5, , , 00000000001.48.5, , , ] 2021-02-11T10:01:44.866 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [00000000001.55.5, , 00000000001.44.5, , 00000000001.59.5, , , 00000000001.48.5, , , ] 2021-02-11T10:01:44.866 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:01:44.859+0000 7f9fb8d3a080 0 bucket checkpoint complete 2021-02-11T10:01:44.895 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-59', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:01:44.895 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-59', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:01:44.896 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-59 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:01:44.947 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:01:44.947 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:01:45.188 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:01:45.182+0000 7f0f8de24080 1 bucket sync caught up with source: 2021-02-11T10:01:45.188 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [00000000001.55.5, , 00000000001.44.5, , 00000000001.59.5, , , 00000000001.48.5, , , ] 2021-02-11T10:01:45.188 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [00000000001.55.5, , 00000000001.44.5, , 00000000001.59.5, , , 00000000001.48.5, , , ] 2021-02-11T10:01:45.188 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:01:45.182+0000 7f0f8de24080 0 bucket checkpoint complete 2021-02-11T10:01:45.223 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-59 zones={test-zone2, test-zone1} 2021-02-11T10:01:45.234 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:01:45.283 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:01:45.283 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:01:45.283 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:01:45.283 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:01:45.283 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:01:45.291 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:01:45.291 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:01:45.291 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:01:45.292 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:01:45.347 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:01:45.403 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:01:45.511 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:01:45.619 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:01:45.670 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:01:45.780 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:01:45.887 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:01:45.944 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:01:46.055 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:01:46.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:01:46.216 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:01:46.328 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:01:46.438 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-59 zones={test-zone2, test-zone1} 2021-02-11T10:01:46.438 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-59 zones={test-zone3, test-zone1} 2021-02-11T10:01:46.447 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:01:46.451 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:01:46.451 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:01:46.451 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:01:46.451 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:01:46.451 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:01:46.455 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:01:46.456 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:01:46.456 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:01:46.456 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:01:46.464 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:01:46.515 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:01:46.628 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:01:46.734 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:01:46.791 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:01:46.903 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:01:47.005 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:01:47.064 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:01:47.175 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:01:47.277 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:01:47.332 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:01:47.442 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:01:47.545 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-59 zones={test-zone3, test-zone1} 2021-02-11T10:01:47.545 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-59 zones={test-zone3, test-zone2} 2021-02-11T10:01:47.556 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:01:47.561 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:01:47.561 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:01:47.561 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:01:47.562 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:01:47.562 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:01:47.567 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:01:47.567 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:01:47.567 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:01:47.567 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:01:47.621 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:01:47.675 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:01:47.787 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:01:47.897 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:01:47.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:01:48.061 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:01:48.165 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:01:48.219 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:01:48.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:01:48.433 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:01:48.486 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:01:48.595 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:01:48.701 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-59 zones={test-zone3, test-zone2} 2021-02-11T10:01:48.702 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:01:48.702 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:01:48.702 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-60 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:01:48.745 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:01:48.745 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:01:48.982 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:01:48.978+0000 7f61a1ba1080 1 bucket sync caught up with source: 2021-02-11T10:01:48.982 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [00000000001.65.5, , 00000000001.45.5, , 00000000001.60.5, , , 00000000001.23.5, , , ] 2021-02-11T10:01:48.982 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [00000000001.65.5, , 00000000001.45.5, , 00000000001.60.5, , , 00000000001.23.5, , , ] 2021-02-11T10:01:48.983 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:01:48.978+0000 7f61a1ba1080 0 bucket checkpoint complete 2021-02-11T10:01:49.018 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:01:49.018 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:01:49.018 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-60 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:01:49.066 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:01:49.066 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:01:49.272 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:01:49.267+0000 7f40c32c1080 1 bucket sync caught up with source: 2021-02-11T10:01:49.272 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [00000000001.65.5, , 00000000001.45.5, , 00000000001.60.5, , , 00000000001.23.5, , , ] 2021-02-11T10:01:49.272 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [00000000001.65.5, , 00000000001.45.5, , 00000000001.60.5, , , 00000000001.23.5, , , ] 2021-02-11T10:01:49.273 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:01:49.267+0000 7f40c32c1080 0 bucket checkpoint complete 2021-02-11T10:01:49.301 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:01:49.302 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:01:49.302 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-60 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:01:49.353 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:01:49.353 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:01:49.555 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:01:49.550+0000 7f5034cb5080 1 bucket sync caught up with source: 2021-02-11T10:01:49.555 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [00000000001.65.5, , 00000000001.45.5, , 00000000001.60.5, , , 00000000001.23.5, , , ] 2021-02-11T10:01:49.555 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [00000000001.65.5, , 00000000001.45.5, , 00000000001.60.5, , , 00000000001.23.5, , , ] 2021-02-11T10:01:49.555 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:01:49.550+0000 7f5034cb5080 0 bucket checkpoint complete 2021-02-11T10:01:49.590 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-60', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:01:49.590 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-60', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:01:49.590 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-60 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:01:49.636 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:01:49.636 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:01:49.857 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:01:49.851+0000 7faa6551b080 1 bucket sync caught up with source: 2021-02-11T10:01:49.857 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [00000000001.47.5, , 00000000001.56.5, , 00000000001.64.5, , , 00000000001.27.5, , , ] 2021-02-11T10:01:49.857 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [00000000001.47.5, , 00000000001.56.5, , 00000000001.64.5, , , 00000000001.27.5, , , ] 2021-02-11T10:01:49.857 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:01:49.851+0000 7faa6551b080 0 bucket checkpoint complete 2021-02-11T10:01:49.894 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-60', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:01:49.894 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-60', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:01:49.894 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-60 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:01:49.939 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:01:49.939 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:01:50.144 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:01:50.139+0000 7f5107936080 1 bucket sync caught up with source: 2021-02-11T10:01:50.145 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [00000000001.47.5, , 00000000001.56.5, , 00000000001.64.5, , , 00000000001.27.5, , , ] 2021-02-11T10:01:50.145 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [00000000001.47.5, , 00000000001.56.5, , 00000000001.64.5, , , 00000000001.27.5, , , ] 2021-02-11T10:01:50.145 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:01:50.139+0000 7f5107936080 0 bucket checkpoint complete 2021-02-11T10:01:50.177 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-60', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:01:50.177 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-60', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:01:50.178 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-60 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:01:50.224 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:01:50.224 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:01:50.438 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:01:50.431+0000 7f02ac76e080 1 bucket sync caught up with source: 2021-02-11T10:01:50.438 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [00000000001.53.5, , 00000000001.47.5, , 00000000001.61.5, , , 00000000001.26.5, , , ] 2021-02-11T10:01:50.438 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [00000000001.53.5, , 00000000001.47.5, , 00000000001.61.5, , , 00000000001.26.5, , , ] 2021-02-11T10:01:50.438 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:01:50.431+0000 7f02ac76e080 0 bucket checkpoint complete 2021-02-11T10:01:50.465 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-60', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:01:50.466 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-60', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:01:50.466 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-60 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:01:50.513 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:01:50.513 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:01:50.751 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:01:50.746+0000 7f127b547080 1 bucket sync caught up with source: 2021-02-11T10:01:50.752 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [00000000001.53.5, , 00000000001.47.5, , 00000000001.61.5, , , 00000000001.26.5, , , ] 2021-02-11T10:01:50.752 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [00000000001.53.5, , 00000000001.47.5, , 00000000001.61.5, , , 00000000001.26.5, , , ] 2021-02-11T10:01:50.752 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:01:50.746+0000 7f127b547080 0 bucket checkpoint complete 2021-02-11T10:01:50.787 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-60 zones={test-zone2, test-zone1} 2021-02-11T10:01:50.798 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:01:50.803 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:01:50.803 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:01:50.803 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:01:50.804 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:01:50.804 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:01:50.809 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:01:50.809 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:01:50.809 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:01:50.809 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:01:50.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:01:50.871 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:01:50.989 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:01:51.094 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:01:51.148 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:01:51.260 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:01:51.364 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:01:51.419 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:01:51.527 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:01:51.629 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:01:51.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:01:51.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:01:51.897 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-60 zones={test-zone2, test-zone1} 2021-02-11T10:01:51.897 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-60 zones={test-zone3, test-zone1} 2021-02-11T10:01:51.907 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:01:51.911 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:01:51.912 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:01:51.912 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:01:51.912 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:01:51.912 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:01:51.916 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:01:51.916 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:01:51.916 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:01:51.917 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:01:51.925 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:01:51.979 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:01:52.091 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:01:52.193 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:01:52.247 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:01:52.355 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:01:52.456 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:01:52.512 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:01:52.623 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:01:52.725 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:01:52.780 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:01:52.891 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:01:52.997 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-60 zones={test-zone3, test-zone1} 2021-02-11T10:01:52.998 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-60 zones={test-zone3, test-zone2} 2021-02-11T10:01:53.009 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:01:53.013 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:01:53.013 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:01:53.013 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:01:53.013 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:01:53.013 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:01:53.019 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:01:53.019 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:01:53.019 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:01:53.019 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:01:53.073 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:01:53.128 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:01:53.239 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:01:53.342 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:01:53.397 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:01:53.508 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:01:53.612 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:01:53.667 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:01:53.779 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:01:53.882 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:01:53.935 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:01:54.042 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:01:54.146 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-60 zones={test-zone3, test-zone2} 2021-02-11T10:01:54.146 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:01:54.146 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:01:54.146 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-61 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:01:54.188 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:01:54.188 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:01:54.439 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:01:54.433+0000 7fe8dffa8080 1 bucket sync caught up with source: 2021-02-11T10:01:54.439 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [00000000001.167.5, , 00000000001.40.5, , 00000000001.32.5, , , 00000000001.61.5, , , ] 2021-02-11T10:01:54.439 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [00000000001.167.5, , 00000000001.40.5, , 00000000001.32.5, , , 00000000001.61.5, , , ] 2021-02-11T10:01:54.439 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:01:54.433+0000 7fe8dffa8080 0 bucket checkpoint complete 2021-02-11T10:01:54.475 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:01:54.476 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:01:54.476 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-61 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:01:54.521 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:01:54.522 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:01:54.719 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:01:54.711+0000 7f81bac6c080 1 bucket sync caught up with source: 2021-02-11T10:01:54.719 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [00000000001.167.5, , 00000000001.40.5, , 00000000001.32.5, , , 00000000001.61.5, , , ] 2021-02-11T10:01:54.719 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [00000000001.167.5, , 00000000001.40.5, , 00000000001.32.5, , , 00000000001.61.5, , , ] 2021-02-11T10:01:54.719 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:01:54.711+0000 7f81bac6c080 0 bucket checkpoint complete 2021-02-11T10:01:54.744 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:01:54.744 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:01:54.744 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-61 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:01:54.790 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:01:54.790 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:01:55.013 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:01:55.005+0000 7fa6583ee080 1 bucket sync caught up with source: 2021-02-11T10:01:55.013 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [00000000001.167.5, , 00000000001.40.5, , 00000000001.32.5, , , 00000000001.61.5, , , ] 2021-02-11T10:01:55.013 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [00000000001.167.5, , 00000000001.40.5, , 00000000001.32.5, , , 00000000001.61.5, , , ] 2021-02-11T10:01:55.013 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:01:55.005+0000 7fa6583ee080 0 bucket checkpoint complete 2021-02-11T10:01:55.046 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-61', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:01:55.046 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-61', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:01:55.046 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-61 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:01:55.093 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:01:55.094 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:01:55.315 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:01:55.311+0000 7fe0f7eb1080 1 bucket sync caught up with source: 2021-02-11T10:01:55.316 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [00000000001.177.5, , 00000000001.43.5, , 00000000001.30.5, , , 00000000001.65.5, , , ] 2021-02-11T10:01:55.316 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [00000000001.177.5, , 00000000001.43.5, , 00000000001.30.5, , , 00000000001.65.5, , , ] 2021-02-11T10:01:55.316 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:01:55.311+0000 7fe0f7eb1080 0 bucket checkpoint complete 2021-02-11T10:01:55.342 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-61', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:01:55.342 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-61', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:01:55.343 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-61 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:01:55.386 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:01:55.386 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:01:55.604 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:01:55.599+0000 7ff35f964080 1 bucket sync caught up with source: 2021-02-11T10:01:55.604 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [00000000001.177.5, , 00000000001.43.5, , 00000000001.30.5, , , 00000000001.65.5, , , ] 2021-02-11T10:01:55.604 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [00000000001.177.5, , 00000000001.43.5, , 00000000001.30.5, , , 00000000001.65.5, , , ] 2021-02-11T10:01:55.604 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:01:55.599+0000 7ff35f964080 0 bucket checkpoint complete 2021-02-11T10:01:55.639 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-61', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:01:55.639 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-61', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:01:55.639 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-61 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:01:55.730 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:01:55.730 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:01:55.954 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:01:55.947+0000 7f4252246080 1 bucket sync caught up with source: 2021-02-11T10:01:55.954 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [00000000001.184.5, , 00000000001.42.5, , 00000000001.31.5, , , 00000000001.88.5, , , ] 2021-02-11T10:01:55.954 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [00000000001.184.5, , 00000000001.42.5, , 00000000001.31.5, , , 00000000001.88.5, , , ] 2021-02-11T10:01:55.955 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:01:55.947+0000 7f4252246080 0 bucket checkpoint complete 2021-02-11T10:01:55.985 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-61', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:01:55.986 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-61', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:01:55.986 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-61 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:01:56.034 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:01:56.034 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:01:56.257 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:01:56.249+0000 7fbe16c1d080 1 bucket sync caught up with source: 2021-02-11T10:01:56.258 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [00000000001.184.5, , 00000000001.42.5, , 00000000001.31.5, , , 00000000001.88.5, , , ] 2021-02-11T10:01:56.258 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [00000000001.184.5, , 00000000001.42.5, , 00000000001.31.5, , , 00000000001.88.5, , , ] 2021-02-11T10:01:56.258 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:01:56.249+0000 7fbe16c1d080 0 bucket checkpoint complete 2021-02-11T10:01:56.292 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-61 zones={test-zone2, test-zone1} 2021-02-11T10:01:56.303 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:01:56.308 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:01:56.308 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:01:56.309 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:01:56.309 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:01:56.309 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:01:56.315 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:01:56.316 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:01:56.316 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:01:56.316 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:01:56.326 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:01:56.378 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:01:56.488 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:01:56.590 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:01:56.643 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:01:56.756 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:01:56.862 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:01:56.915 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:01:57.026 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:01:57.133 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:01:57.188 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:01:57.303 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:01:57.405 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-61 zones={test-zone2, test-zone1} 2021-02-11T10:01:57.406 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-61 zones={test-zone3, test-zone1} 2021-02-11T10:01:57.415 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:01:57.420 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:01:57.420 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:01:57.420 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:01:57.420 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:01:57.420 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:01:57.424 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:01:57.424 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:01:57.424 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:01:57.424 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:01:57.433 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:01:57.489 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:01:57.599 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:01:57.701 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:01:57.756 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:01:57.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:01:57.970 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:01:58.039 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:01:58.164 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:01:58.275 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:01:58.334 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:01:58.442 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:01:58.545 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-61 zones={test-zone3, test-zone1} 2021-02-11T10:01:58.545 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-61 zones={test-zone3, test-zone2} 2021-02-11T10:01:58.555 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:01:58.560 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:01:58.560 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:01:58.560 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:01:58.560 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:01:58.560 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:01:58.606 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:01:58.607 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:01:58.607 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:01:58.607 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:01:58.620 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:01:58.671 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:01:58.779 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:01:58.881 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:01:58.940 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:01:59.051 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:01:59.153 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:01:59.208 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:01:59.319 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:01:59.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:01:59.467 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:01:59.579 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:01:59.685 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-61 zones={test-zone3, test-zone2} 2021-02-11T10:01:59.685 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'tkscyf-59', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:01:59.686 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'tkscyf-59', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:01:59.686 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket tkscyf-59 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T10:01:59.728 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:01:59.728 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:02:00.031 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'tkscyf-59', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:02:00.031 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'tkscyf-59', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:02:00.031 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket tkscyf-59 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T10:02:00.114 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:02:00.114 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:02:00.418 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'tkscyf-60', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:02:00.418 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'tkscyf-60', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:02:00.418 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket tkscyf-60 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T10:02:00.498 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:02:00.498 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:02:00.794 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'tkscyf-60', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:02:00.794 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'tkscyf-60', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:02:00.794 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket tkscyf-60 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T10:02:00.876 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:02:00.876 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:02:01.178 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'tkscyf-61', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:02:01.178 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'tkscyf-61', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:02:01.179 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket tkscyf-61 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T10:02:01.259 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:02:01.259 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:02:01.558 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'tkscyf-61', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:02:01.558 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'tkscyf-61', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:02:01.558 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket tkscyf-61 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T10:02:01.641 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:02:01.641 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:02:02.523 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:02:02.523 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:02:02.523 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-59 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:02:02.571 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:02:02.571 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:02:02.829 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:02:02.821+0000 7fa98dcbd080 1 waiting for incremental sync to catch up: 2021-02-11T10:02:02.829 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [00000000006.79.5, , 00000000006.47.5, 00000000004.171.1, 00000000006.59.5, , 00000000004.36.1, 00000000006.53.5, 00000000004.48.1, , 00000000004.73.1] 2021-02-11T10:02:02.829 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [00000000006.79.5, 00000000004.27.1, 00000000006.47.5, 00000000004.171.1, 00000000006.59.5, 00000000004.65.1, 00000000004.36.1, 00000000006.53.5, 00000000004.48.1, 00000000004.58.1, 00000000004.73.1] 2021-02-11T10:02:07.833 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:02:07.825+0000 7fa98dcbd080 1 waiting for incremental sync to catch up: 2021-02-11T10:02:07.834 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [00000000006.79.5, , 00000000006.47.5, 00000000004.171.1, 00000000006.59.5, , 00000000004.36.1, 00000000006.53.5, 00000000004.48.1, , 00000000004.73.1] 2021-02-11T10:02:07.834 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [00000000006.79.5, 00000000004.27.1, 00000000006.47.5, 00000000004.171.1, 00000000006.59.5, 00000000004.65.1, 00000000004.36.1, 00000000006.53.5, 00000000004.48.1, 00000000004.58.1, 00000000004.73.1] 2021-02-11T10:02:12.838 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:02:12.829+0000 7fa98dcbd080 1 waiting for incremental sync to catch up: 2021-02-11T10:02:19.827 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [00000000006.79.5, , 00000000006.47.5, 00000000004.171.1, 00000000006.59.5, , 00000000004.36.1, 00000000006.53.5, 00000000004.48.1, , 00000000004.73.1] 2021-02-11T10:02:19.828 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [00000000006.79.5, 00000000004.27.1, 00000000006.47.5, 00000000004.171.1, 00000000006.59.5, 00000000004.65.1, 00000000004.36.1, 00000000006.53.5, 00000000004.48.1, 00000000004.58.1, 00000000004.73.1] 2021-02-11T10:02:19.829 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:02:17.833+0000 7fa98dcbd080 1 bucket sync caught up with source: 2021-02-11T10:02:19.829 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [00000000006.79.5, 00000000004.27.1, 00000000006.47.5, 00000000004.171.1, 00000000006.59.5, 00000000004.65.1, 00000000004.36.1, 00000000006.53.5, 00000000004.48.1, 00000000004.58.1, 00000000004.73.1] 2021-02-11T10:02:19.829 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [00000000006.79.5, 00000000004.27.1, 00000000006.47.5, 00000000004.171.1, 00000000006.59.5, 00000000004.65.1, 00000000004.36.1, 00000000006.53.5, 00000000004.48.1, 00000000004.58.1, 00000000004.73.1] 2021-02-11T10:02:19.829 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:02:17.833+0000 7fa98dcbd080 0 bucket checkpoint complete 2021-02-11T10:02:19.830 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:02:19.830 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:02:19.830 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-59 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:02:19.876 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:02:19.876 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:02:20.088 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:02:20.082+0000 7f3bc3c38080 1 bucket sync caught up with source: 2021-02-11T10:02:20.088 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [00000000006.79.5, 00000000004.27.1, 00000000006.47.5, 00000000004.171.1, 00000000006.59.5, 00000000004.65.1, 00000000004.36.1, 00000000006.53.5, 00000000004.48.1, 00000000004.58.1, 00000000004.73.1] 2021-02-11T10:02:20.088 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [00000000006.79.5, 00000000004.27.1, 00000000006.47.5, 00000000004.171.1, 00000000006.59.5, 00000000004.65.1, 00000000004.36.1, 00000000006.53.5, 00000000004.48.1, 00000000004.58.1, 00000000004.73.1] 2021-02-11T10:02:20.088 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:02:20.082+0000 7f3bc3c38080 0 bucket checkpoint complete 2021-02-11T10:02:20.118 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:02:20.118 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:02:20.119 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-59 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:02:20.166 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:02:20.167 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:02:20.393 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:02:20.385+0000 7f75b8bb4080 1 bucket sync caught up with source: 2021-02-11T10:02:20.393 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [00000000006.79.5, 00000000004.27.1, 00000000006.47.5, 00000000004.171.1, 00000000006.59.5, 00000000004.65.1, 00000000004.36.1, 00000000006.53.5, 00000000004.48.1, 00000000004.58.1, 00000000004.73.1] 2021-02-11T10:02:20.393 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [00000000006.79.5, 00000000004.27.1, 00000000006.47.5, 00000000004.171.1, 00000000006.59.5, 00000000004.65.1, 00000000004.36.1, 00000000006.53.5, 00000000004.48.1, 00000000004.58.1, 00000000004.73.1] 2021-02-11T10:02:20.393 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:02:20.385+0000 7f75b8bb4080 0 bucket checkpoint complete 2021-02-11T10:02:20.426 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-59', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:02:20.426 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-59', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:02:20.427 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-59 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:02:20.474 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:02:20.474 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:02:20.699 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:02:20.694+0000 7f34be0f1080 1 bucket sync caught up with source: 2021-02-11T10:02:20.699 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [00000000002.51.5, , 00000000002.62.5, , 00000000002.58.5, , , 00000000002.66.5, , , ] 2021-02-11T10:02:20.699 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [00000000002.51.5, , 00000000002.62.5, , 00000000002.58.5, , , 00000000002.66.5, , , ] 2021-02-11T10:02:20.699 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:02:20.694+0000 7f34be0f1080 0 bucket checkpoint complete 2021-02-11T10:02:20.736 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-59', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:02:20.736 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-59', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:02:20.736 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-59 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:02:20.783 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:02:20.783 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:02:21.007 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:02:21.002+0000 7fe820604080 1 bucket sync caught up with source: 2021-02-11T10:02:21.007 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [00000000002.51.5, , 00000000002.62.5, , 00000000002.58.5, , , 00000000002.66.5, , , ] 2021-02-11T10:02:21.007 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [00000000002.51.5, , 00000000002.62.5, , 00000000002.58.5, , , 00000000002.66.5, , , ] 2021-02-11T10:02:21.007 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:02:21.002+0000 7fe820604080 0 bucket checkpoint complete 2021-02-11T10:02:21.042 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-59', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:02:21.042 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-59', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:02:21.042 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-59 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:02:21.088 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:02:21.089 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:02:21.314 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:02:21.306+0000 7f5430ff6080 1 bucket sync caught up with source: 2021-02-11T10:02:21.314 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [00000000002.63.5, , 00000000002.48.5, , 00000000002.63.5, , , 00000000002.54.5, , , ] 2021-02-11T10:02:21.315 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [00000000002.63.5, , 00000000002.48.5, , 00000000002.63.5, , , 00000000002.54.5, , , ] 2021-02-11T10:02:21.315 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:02:21.306+0000 7f5430ff6080 0 bucket checkpoint complete 2021-02-11T10:02:21.344 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-59', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:02:21.345 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-59', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:02:21.345 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-59 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:02:21.390 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:02:21.390 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:02:21.626 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:02:21.617+0000 7fdb88b4f080 1 bucket sync caught up with source: 2021-02-11T10:02:21.626 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [00000000002.63.5, , 00000000002.48.5, , 00000000002.63.5, , , 00000000002.54.5, , , ] 2021-02-11T10:02:21.626 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [00000000002.63.5, , 00000000002.48.5, , 00000000002.63.5, , , 00000000002.54.5, , , ] 2021-02-11T10:02:21.626 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:02:21.617+0000 7fdb88b4f080 0 bucket checkpoint complete 2021-02-11T10:02:21.661 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-59 zones={test-zone2, test-zone1} 2021-02-11T10:02:21.672 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:02:21.679 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:02:21.679 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:02:21.680 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:02:21.680 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:02:21.680 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T10:02:21.680 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T10:02:21.680 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T10:02:21.680 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T10:02:21.680 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:02:21.686 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:02:21.686 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:02:21.686 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:02:21.686 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:02:21.687 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T10:02:21.687 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T10:02:21.687 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T10:02:21.687 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T10:02:21.698 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:02:21.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:02:21.859 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:02:21.965 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:02:22.020 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:02:22.131 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:02:22.234 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:02:22.288 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:02:22.400 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:02:22.506 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:02:22.559 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:02:22.671 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:02:22.773 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:02:22.827 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:02:22.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:02:23.042 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:02:23.095 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:02:23.208 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:02:23.311 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:02:23.367 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:02:23.474 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:02:23.578 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:02:23.631 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:02:23.738 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:02:23.841 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-59 zones={test-zone2, test-zone1} 2021-02-11T10:02:23.842 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-59 zones={test-zone3, test-zone1} 2021-02-11T10:02:23.851 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:02:23.856 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:02:23.856 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:02:23.856 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:02:23.856 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:02:23.856 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T10:02:23.856 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T10:02:23.856 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T10:02:23.856 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T10:02:23.857 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:02:23.862 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:02:23.862 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:02:23.862 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:02:23.862 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:02:23.862 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T10:02:23.862 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T10:02:23.862 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T10:02:23.862 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T10:02:23.871 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:02:23.923 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:02:24.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:02:24.133 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:02:24.188 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:02:24.299 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:02:24.401 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:02:24.456 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:02:24.566 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:02:24.669 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:02:24.727 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:02:24.835 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:02:24.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:02:24.991 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:02:25.103 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:02:25.205 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:02:25.260 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:02:25.369 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:02:25.474 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:02:25.527 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:02:25.638 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:02:25.740 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:02:25.796 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:02:25.906 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:02:26.009 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-59 zones={test-zone3, test-zone1} 2021-02-11T10:02:26.009 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-59 zones={test-zone3, test-zone2} 2021-02-11T10:02:26.019 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:02:26.024 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:02:26.024 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:02:26.024 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:02:26.024 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:02:26.024 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T10:02:26.025 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T10:02:26.025 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T10:02:26.025 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T10:02:26.025 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:02:26.031 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:02:26.031 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:02:26.031 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:02:26.031 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:02:26.031 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T10:02:26.031 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T10:02:26.031 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T10:02:26.031 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T10:02:26.042 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:02:26.095 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:02:26.202 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:02:26.309 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:02:26.361 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:02:26.476 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:02:26.582 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:02:26.636 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:02:26.747 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:02:26.853 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:02:26.907 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:02:27.018 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:02:27.122 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:02:27.175 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:02:27.287 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:02:27.383 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:02:27.435 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:02:27.547 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:02:27.650 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:02:27.704 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:02:27.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:02:27.921 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:02:27.975 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:02:28.084 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:02:28.189 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-59 zones={test-zone3, test-zone2} 2021-02-11T10:02:28.189 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:02:28.189 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:02:28.189 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-60 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:02:28.236 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:02:28.236 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:02:28.451 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:02:28.445+0000 7f373b616080 1 bucket sync caught up with source: 2021-02-11T10:02:28.452 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [00000000006.81.5, 00000000004.42.1, 00000000006.51.5, 00000000004.26.1, 00000000006.66.5, 00000000004.51.1, 00000000004.42.1, 00000000006.29.5, 00000000004.52.1, 00000000004.175.1, 00000000004.40.1] 2021-02-11T10:02:28.452 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [00000000006.81.5, 00000000004.42.1, 00000000006.51.5, 00000000004.26.1, 00000000006.66.5, 00000000004.51.1, 00000000004.42.1, 00000000006.29.5, 00000000004.52.1, 00000000004.175.1, 00000000004.40.1] 2021-02-11T10:02:28.452 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:02:28.445+0000 7f373b616080 0 bucket checkpoint complete 2021-02-11T10:02:28.485 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:02:28.485 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:02:28.485 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-60 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:02:28.532 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:02:28.533 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:02:28.752 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:02:28.746+0000 7f59cd8c2080 1 bucket sync caught up with source: 2021-02-11T10:02:28.752 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [00000000006.81.5, 00000000004.42.1, 00000000006.51.5, 00000000004.26.1, 00000000006.66.5, 00000000004.51.1, 00000000004.42.1, 00000000006.29.5, 00000000004.52.1, 00000000004.175.1, 00000000004.40.1] 2021-02-11T10:02:28.752 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [00000000006.81.5, 00000000004.42.1, 00000000006.51.5, 00000000004.26.1, 00000000006.66.5, 00000000004.51.1, 00000000004.42.1, 00000000006.29.5, 00000000004.52.1, 00000000004.175.1, 00000000004.40.1] 2021-02-11T10:02:28.752 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:02:28.746+0000 7f59cd8c2080 0 bucket checkpoint complete 2021-02-11T10:02:28.779 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:02:28.779 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:02:28.779 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-60 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:02:28.828 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:02:28.829 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:02:29.067 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:02:29.061+0000 7f90bd9c3080 1 bucket sync caught up with source: 2021-02-11T10:02:29.067 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [00000000006.81.5, 00000000004.42.1, 00000000006.51.5, 00000000004.26.1, 00000000006.66.5, 00000000004.51.1, 00000000004.42.1, 00000000006.29.5, 00000000004.52.1, 00000000004.175.1, 00000000004.40.1] 2021-02-11T10:02:29.067 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [00000000006.81.5, 00000000004.42.1, 00000000006.51.5, 00000000004.26.1, 00000000006.66.5, 00000000004.51.1, 00000000004.42.1, 00000000006.29.5, 00000000004.52.1, 00000000004.175.1, 00000000004.40.1] 2021-02-11T10:02:29.067 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:02:29.061+0000 7f90bd9c3080 0 bucket checkpoint complete 2021-02-11T10:02:29.099 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-60', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:02:29.099 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-60', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:02:29.099 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-60 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:02:29.147 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:02:29.147 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:02:29.359 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:02:29.354+0000 7fa7b28f1080 1 bucket sync caught up with source: 2021-02-11T10:02:29.359 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [00000000002.53.5, , 00000000002.58.5, , 00000000002.66.5, , , 00000000002.29.5, , , ] 2021-02-11T10:02:29.359 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [00000000002.53.5, , 00000000002.58.5, , 00000000002.66.5, , , 00000000002.29.5, , , ] 2021-02-11T10:02:29.359 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:02:29.354+0000 7fa7b28f1080 0 bucket checkpoint complete 2021-02-11T10:02:29.392 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-60', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:02:29.392 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-60', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:02:29.392 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-60 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:02:29.438 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:02:29.438 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:02:29.664 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:02:29.658+0000 7fa835e26080 1 bucket sync caught up with source: 2021-02-11T10:02:29.664 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [00000000002.53.5, , 00000000002.58.5, , 00000000002.66.5, , , 00000000002.29.5, , , ] 2021-02-11T10:02:29.664 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [00000000002.53.5, , 00000000002.58.5, , 00000000002.66.5, , , 00000000002.29.5, , , ] 2021-02-11T10:02:29.664 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:02:29.658+0000 7fa835e26080 0 bucket checkpoint complete 2021-02-11T10:02:29.698 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-60', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:02:29.698 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-60', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:02:29.699 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-60 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:02:29.790 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:02:29.790 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:02:30.012 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:02:30.006+0000 7f2f3676f080 1 bucket sync caught up with source: 2021-02-11T10:02:30.012 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [00000000002.65.5, , 00000000002.51.5, , 00000000002.65.5, , , 00000000002.30.5, , , ] 2021-02-11T10:02:30.012 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [00000000002.65.5, , 00000000002.51.5, , 00000000002.65.5, , , 00000000002.30.5, , , ] 2021-02-11T10:02:30.012 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:02:30.006+0000 7f2f3676f080 0 bucket checkpoint complete 2021-02-11T10:02:30.040 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-60', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:02:30.041 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-60', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:02:30.041 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-60 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:02:30.087 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:02:30.087 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:02:30.311 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:02:30.305+0000 7fd043865080 1 bucket sync caught up with source: 2021-02-11T10:02:30.311 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [00000000002.65.5, , 00000000002.51.5, , 00000000002.65.5, , , 00000000002.30.5, , , ] 2021-02-11T10:02:30.311 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [00000000002.65.5, , 00000000002.51.5, , 00000000002.65.5, , , 00000000002.30.5, , , ] 2021-02-11T10:02:30.311 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:02:30.305+0000 7fd043865080 0 bucket checkpoint complete 2021-02-11T10:02:30.344 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-60 zones={test-zone2, test-zone1} 2021-02-11T10:02:30.355 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:02:30.363 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:02:30.363 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:02:30.363 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:02:30.363 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:02:30.363 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T10:02:30.363 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T10:02:30.363 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T10:02:30.363 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T10:02:30.363 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:02:30.368 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:02:30.369 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:02:30.369 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:02:30.369 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:02:30.369 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T10:02:30.369 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T10:02:30.369 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T10:02:30.369 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T10:02:30.381 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:02:30.435 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:02:30.541 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:02:30.646 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:02:30.699 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:02:30.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:02:30.914 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:02:30.967 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:02:31.075 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:02:31.178 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:02:31.231 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:02:31.335 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:02:31.438 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:02:31.491 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:02:31.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:02:31.705 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:02:31.759 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:02:31.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:02:31.969 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:02:32.023 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:02:32.135 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:02:32.241 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:02:32.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:02:32.404 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:02:32.509 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-60 zones={test-zone2, test-zone1} 2021-02-11T10:02:32.510 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-60 zones={test-zone3, test-zone1} 2021-02-11T10:02:32.519 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:02:32.524 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:02:32.524 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:02:32.524 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:02:32.524 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:02:32.524 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T10:02:32.524 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T10:02:32.524 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T10:02:32.524 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T10:02:32.524 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:02:32.528 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:02:32.528 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:02:32.529 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:02:32.529 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:02:32.529 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T10:02:32.529 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T10:02:32.529 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T10:02:32.529 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T10:02:32.538 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:02:32.590 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:02:32.698 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:02:32.800 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:02:32.856 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:02:32.967 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:02:33.070 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:02:33.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:02:33.230 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:02:33.334 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:02:33.390 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:02:33.498 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:02:33.600 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:02:33.656 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:02:33.767 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:02:33.869 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:02:33.923 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:02:34.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:02:34.137 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:02:34.192 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:02:34.307 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:02:34.409 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:02:34.464 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:02:34.573 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:02:34.676 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-60 zones={test-zone3, test-zone1} 2021-02-11T10:02:34.676 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-60 zones={test-zone3, test-zone2} 2021-02-11T10:02:34.687 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:02:34.692 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:02:34.693 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:02:34.693 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:02:34.693 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:02:34.693 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T10:02:34.693 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T10:02:34.693 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T10:02:34.693 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T10:02:34.693 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:02:34.699 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:02:34.699 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:02:34.699 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:02:34.699 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:02:34.699 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T10:02:34.699 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T10:02:34.699 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T10:02:34.699 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T10:02:34.710 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:02:34.763 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:02:34.872 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:02:34.978 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:02:35.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:02:35.144 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:02:35.249 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:02:35.309 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:02:35.418 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:02:35.520 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:02:35.576 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:02:35.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:02:35.789 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:02:35.844 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:02:35.957 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:02:36.062 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:02:36.116 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:02:36.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:02:36.334 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:02:36.392 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:02:36.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:02:36.609 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:02:36.663 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:02:36.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:02:36.878 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-60 zones={test-zone3, test-zone2} 2021-02-11T10:02:36.878 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:02:36.878 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:02:36.878 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-61 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:02:36.922 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:02:36.922 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:02:37.165 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:02:37.157+0000 7f3b5ae59080 1 bucket sync caught up with source: 2021-02-11T10:02:37.165 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [00000000006.181.5, 00000000004.55.1, 00000000006.46.5, 00000000004.31.1, 00000000006.42.5, 00000000004.43.1, 00000000004.44.1, 00000000006.71.5, 00000000004.53.1, 00000000004.62.1, 00000000004.51.1] 2021-02-11T10:02:37.166 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [00000000006.181.5, 00000000004.55.1, 00000000006.46.5, 00000000004.31.1, 00000000006.42.5, 00000000004.43.1, 00000000004.44.1, 00000000006.71.5, 00000000004.53.1, 00000000004.62.1, 00000000004.51.1] 2021-02-11T10:02:37.166 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:02:37.157+0000 7f3b5ae59080 0 bucket checkpoint complete 2021-02-11T10:02:37.199 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:02:37.199 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:02:37.199 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-61 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:02:37.246 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:02:37.246 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:02:37.449 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:02:37.442+0000 7f2a737d6080 1 bucket sync caught up with source: 2021-02-11T10:02:37.450 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [00000000006.181.5, 00000000004.55.1, 00000000006.46.5, 00000000004.31.1, 00000000006.42.5, 00000000004.43.1, 00000000004.44.1, 00000000006.71.5, 00000000004.53.1, 00000000004.62.1, 00000000004.51.1] 2021-02-11T10:02:37.450 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [00000000006.181.5, 00000000004.55.1, 00000000006.46.5, 00000000004.31.1, 00000000006.42.5, 00000000004.43.1, 00000000004.44.1, 00000000006.71.5, 00000000004.53.1, 00000000004.62.1, 00000000004.51.1] 2021-02-11T10:02:37.450 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:02:37.442+0000 7f2a737d6080 0 bucket checkpoint complete 2021-02-11T10:02:37.479 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:02:37.479 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:02:37.479 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-61 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:02:37.526 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:02:37.526 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:02:37.744 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:02:37.737+0000 7ff958453080 1 bucket sync caught up with source: 2021-02-11T10:02:37.744 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [00000000006.181.5, 00000000004.55.1, 00000000006.46.5, 00000000004.31.1, 00000000006.42.5, 00000000004.43.1, 00000000004.44.1, 00000000006.71.5, 00000000004.53.1, 00000000004.62.1, 00000000004.51.1] 2021-02-11T10:02:37.744 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [00000000006.181.5, 00000000004.55.1, 00000000006.46.5, 00000000004.31.1, 00000000006.42.5, 00000000004.43.1, 00000000004.44.1, 00000000006.71.5, 00000000004.53.1, 00000000004.62.1, 00000000004.51.1] 2021-02-11T10:02:37.744 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:02:37.737+0000 7ff958453080 0 bucket checkpoint complete 2021-02-11T10:02:37.778 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-61', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:02:37.778 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-61', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:02:37.778 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-61 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:02:37.827 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:02:37.827 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:02:38.061 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:02:38.054+0000 7fe71eefc080 1 bucket sync caught up with source: 2021-02-11T10:02:38.061 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [00000000002.185.5, , 00000000002.45.5, , 00000000002.32.5, , , 00000000002.71.5, , , ] 2021-02-11T10:02:38.061 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [00000000002.185.5, , 00000000002.45.5, , 00000000002.32.5, , , 00000000002.71.5, , , ] 2021-02-11T10:02:38.061 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:02:38.054+0000 7fe71eefc080 0 bucket checkpoint complete 2021-02-11T10:02:38.096 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-61', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:02:38.096 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-61', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:02:38.096 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-61 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:02:38.139 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:02:38.139 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:02:38.358 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:02:38.350+0000 7fa1e38c3080 1 bucket sync caught up with source: 2021-02-11T10:02:38.358 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [00000000002.185.5, , 00000000002.45.5, , 00000000002.32.5, , , 00000000002.71.5, , , ] 2021-02-11T10:02:38.358 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [00000000002.185.5, , 00000000002.45.5, , 00000000002.32.5, , , 00000000002.71.5, , , ] 2021-02-11T10:02:38.358 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:02:38.350+0000 7fa1e38c3080 0 bucket checkpoint complete 2021-02-11T10:02:38.389 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-61', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:02:38.389 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-61', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:02:38.389 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-61 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:02:38.478 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:02:38.478 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:02:38.705 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:02:38.698+0000 7fcb237f4080 1 bucket sync caught up with source: 2021-02-11T10:02:38.705 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [00000000002.192.5, , 00000000002.46.5, , 00000000002.37.5, , , 00000000002.94.5, , , ] 2021-02-11T10:02:38.705 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [00000000002.192.5, , 00000000002.46.5, , 00000000002.37.5, , , 00000000002.94.5, , , ] 2021-02-11T10:02:38.705 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:02:38.698+0000 7fcb237f4080 0 bucket checkpoint complete 2021-02-11T10:02:38.736 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-61', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:02:38.736 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-61', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:02:38.736 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-61 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:02:38.784 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:02:38.784 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:02:39.017 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:02:39.009+0000 7f9fa39f2080 1 bucket sync caught up with source: 2021-02-11T10:02:39.017 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [00000000002.192.5, , 00000000002.46.5, , 00000000002.37.5, , , 00000000002.94.5, , , ] 2021-02-11T10:02:39.017 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [00000000002.192.5, , 00000000002.46.5, , 00000000002.37.5, , , 00000000002.94.5, , , ] 2021-02-11T10:02:39.017 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:02:39.009+0000 7f9fa39f2080 0 bucket checkpoint complete 2021-02-11T10:02:39.052 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-61 zones={test-zone2, test-zone1} 2021-02-11T10:02:39.062 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:02:39.068 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:02:39.068 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:02:39.068 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:02:39.068 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:02:39.069 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T10:02:39.069 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T10:02:39.069 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T10:02:39.069 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T10:02:39.069 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:02:39.075 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:02:39.075 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:02:39.076 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:02:39.076 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:02:39.076 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T10:02:39.076 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T10:02:39.076 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T10:02:39.076 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T10:02:39.087 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:02:39.139 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:02:39.252 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:02:39.355 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:02:39.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:02:39.524 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:02:39.629 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:02:39.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:02:39.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:02:39.898 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:02:39.956 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:02:40.067 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:02:40.170 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:02:40.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:02:40.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:02:40.433 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:02:40.487 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:02:40.600 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:02:40.705 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:02:40.759 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:02:40.866 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:02:40.969 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:02:41.023 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:02:41.135 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:02:41.241 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-61 zones={test-zone2, test-zone1} 2021-02-11T10:02:41.242 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-61 zones={test-zone3, test-zone1} 2021-02-11T10:02:41.250 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:02:41.254 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:02:41.254 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:02:41.255 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:02:41.255 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:02:41.255 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T10:02:41.255 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T10:02:41.255 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T10:02:41.255 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T10:02:41.255 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:02:41.261 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:02:41.261 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:02:41.261 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:02:41.261 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:02:41.261 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T10:02:41.261 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T10:02:41.261 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T10:02:41.261 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T10:02:41.271 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:02:41.322 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:02:41.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:02:41.534 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:02:41.587 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:02:41.695 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:02:41.798 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:02:41.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:02:41.963 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:02:42.065 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:02:42.120 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:02:42.230 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:02:42.333 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:02:42.387 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:02:42.495 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:02:42.598 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:02:42.652 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:02:42.762 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:02:42.864 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:02:42.920 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:02:43.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:02:43.133 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:02:43.188 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:02:43.296 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:02:43.401 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-61 zones={test-zone3, test-zone1} 2021-02-11T10:02:43.402 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-61 zones={test-zone3, test-zone2} 2021-02-11T10:02:43.412 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:02:43.416 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:02:43.416 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:02:43.416 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:02:43.416 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:02:43.416 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T10:02:43.417 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T10:02:43.417 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T10:02:43.417 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T10:02:43.417 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:02:43.422 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:02:43.422 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:02:43.422 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:02:43.422 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:02:43.422 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T10:02:43.422 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T10:02:43.422 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T10:02:43.423 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T10:02:43.432 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:02:43.487 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:02:43.599 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:02:43.701 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:02:43.756 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:02:43.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:02:43.973 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:02:44.027 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:02:44.139 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:02:47.689 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:02:47.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:02:47.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:02:47.958 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:02:48.012 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:02:48.124 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:02:48.229 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:02:48.284 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:02:48.391 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:02:48.497 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:02:48.551 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:02:48.663 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:02:48.765 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:02:48.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:02:48.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:02:49.037 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-61 zones={test-zone3, test-zone2} 2021-02-11T10:02:49.038 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:02:49.038 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:02:49.038 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:02:49.079 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:02:49.079 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:02:49.308 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613037721.927274_563.3","last_update":"2021-02-11T10:02:01.927274Z"},{"marker":"1_1613037722.466799_504.1","last_update":"2021-02-11T10:02:02.466799Z"},{"marker":"1_1613037722.696491_250.1","last_update":"2021-02-11T10:02:02.696491Z"},{"marker":"1_1613037722.756107_327.1","last_update":"2021-02-11T10:02:02.756107Z"}] 2021-02-11T10:02:49.308 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-02-11T10:02:49.308 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:02:49.308 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:02:49.308 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:02:49.351 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:02:49.351 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:02:49.607 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4703921989677143155},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037721.927274_563.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:02:01.927274Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037722.466799_504.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:02:02.466799Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613037722.696491_250.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:02:02.696491Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037722.756107_327.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:02:02.756107Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:02:49.607 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-02-11T10:02:49.607 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:02:49.607 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:02:49.607 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:02:49.647 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:02:49.648 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:02:49.888 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613037721.927274_563.3","last_update":"2021-02-11T10:02:01.927274Z"},{"marker":"1_1613037722.466799_504.1","last_update":"2021-02-11T10:02:02.466799Z"},{"marker":"1_1613037722.696491_250.1","last_update":"2021-02-11T10:02:02.696491Z"},{"marker":"1_1613037722.756107_327.1","last_update":"2021-02-11T10:02:02.756107Z"}] 2021-02-11T10:02:49.888 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-02-11T10:02:49.888 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:02:49.888 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:02:49.888 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:02:49.973 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:02:49.973 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:02:50.487 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":13312279424886386302},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037721.927274_563.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:02:01.927274Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037722.466799_504.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:02:02.466799Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613037722.696491_250.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:02:02.696491Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037722.756107_327.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:02:02.756107Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:02:50.487 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-02-11T10:02:50.487 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:02:50.487 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:02:50.488 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:02:50.569 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:02:50.569 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:02:50.809 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613037721.927274_563.3","last_update":"2021-02-11T10:02:01.927274Z"},{"marker":"1_1613037722.466799_504.1","last_update":"2021-02-11T10:02:02.466799Z"},{"marker":"1_1613037722.696491_250.1","last_update":"2021-02-11T10:02:02.696491Z"},{"marker":"1_1613037722.756107_327.1","last_update":"2021-02-11T10:02:02.756107Z"}] 2021-02-11T10:02:50.809 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-02-11T10:02:50.809 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:02:50.809 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:02:50.810 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:02:50.852 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:02:50.853 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:02:51.104 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2896161496309628191},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037721.927274_563.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:02:01.927274Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037722.466799_504.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:02:02.466799Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613037722.696491_250.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:02:02.696491Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037722.756107_327.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:02:02.756107Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:02:51.105 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-02-11T10:02:51.105 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:02:51.105 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:02:51.105 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:02:51.184 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:02:51.185 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:02:51.438 INFO:teuthology.orchestra.run.smithi098.stdout:[{"marker":"1_1613037662.670057_236.1","last_update":"2021-02-11T10:01:02.670057Z"},{"marker":"1_1613037722.504256_493.1","last_update":"2021-02-11T10:02:02.504256Z"},{"marker":"1_1613037722.740491_263.1","last_update":"2021-02-11T10:02:02.740491Z"},{"marker":"1_1613037722.719639_342.1","last_update":"2021-02-11T10:02:02.719639Z"}] 2021-02-11T10:02:51.439 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-02-11T10:02:51.439 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:02:51.439 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:02:51.439 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:02:51.482 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:02:51.482 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:02:51.730 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":15226599365335882107},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037662.670057_236.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:01:02.670057Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037722.504256_493.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:02:02.504256Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613037722.740491_263.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:02:02.740491Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037722.719639_342.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:02:02.719639Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:02:51.730 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-02-11T10:02:51.731 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:02:51.731 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:02:51.731 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:02:51.771 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:02:51.771 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:02:52.048 INFO:teuthology.orchestra.run.smithi098.stdout:[{"marker":"1_1613037662.670057_236.1","last_update":"2021-02-11T10:01:02.670057Z"},{"marker":"1_1613037722.504256_493.1","last_update":"2021-02-11T10:02:02.504256Z"},{"marker":"1_1613037722.740491_263.1","last_update":"2021-02-11T10:02:02.740491Z"},{"marker":"1_1613037722.719639_342.1","last_update":"2021-02-11T10:02:02.719639Z"}] 2021-02-11T10:02:52.048 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-02-11T10:02:52.049 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:02:52.049 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:02:52.049 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:02:52.090 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:02:52.090 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:02:52.336 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":10999913493642558030},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037662.670057_236.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:01:02.670057Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037722.504256_493.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:02:02.504256Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613037722.740491_263.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:02:02.740491Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037722.719639_342.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:02:02.719639Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:02:52.336 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-02-11T10:02:52.336 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:02:52.336 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:02:52.337 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:02:52.416 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:02:52.416 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:02:52.653 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613037662.065493_287.1","last_update":"2021-02-11T10:01:02.065493Z"},{"marker":"1_1613037722.391333_406.1","last_update":"2021-02-11T10:02:02.391333Z"},{"marker":"1_1613037722.490645_180.1","last_update":"2021-02-11T10:02:02.490645Z"},{"marker":"1_1613037722.697980_248.1","last_update":"2021-02-11T10:02:02.697980Z"}] 2021-02-11T10:02:52.653 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-02-11T10:02:52.654 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:02:52.654 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:02:52.654 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:02:52.736 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:02:52.736 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:02:52.979 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2753764877348348355},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037662.065493_287.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:01:02.065493Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037722.391333_406.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:02:02.391333Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613037722.490645_180.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:02:02.490645Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037722.697980_248.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:02:02.697980Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:02:52.979 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-02-11T10:02:52.979 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:02:52.979 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:02:52.980 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:02:53.061 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:02:53.061 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:02:53.299 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613037662.065493_287.1","last_update":"2021-02-11T10:01:02.065493Z"},{"marker":"1_1613037722.391333_406.1","last_update":"2021-02-11T10:02:02.391333Z"},{"marker":"1_1613037722.490645_180.1","last_update":"2021-02-11T10:02:02.490645Z"},{"marker":"1_1613037722.697980_248.1","last_update":"2021-02-11T10:02:02.697980Z"}] 2021-02-11T10:02:53.299 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-02-11T10:02:53.299 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:02:53.300 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:02:53.300 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:02:53.346 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:02:53.346 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:02:53.585 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":7143279782539285748},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037662.065493_287.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:01:02.065493Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037722.391333_406.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:02:02.391333Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613037722.490645_180.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:02:02.490645Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037722.697980_248.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:02:02.697980Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:02:53.586 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-02-11T10:02:53.587 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_sync_enable_right_after_disable ... ok 2021-02-11T10:02:53.587 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=tkscyf-62 2021-02-11T10:02:53.628 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=tkscyf-63 2021-02-11T10:02:53.717 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=tkscyf-64 2021-02-11T10:02:54.358 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:02:54.359 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:02:54.359 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:02:54.401 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:02:54.401 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:02:54.601 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 2021-02-11T10:02:54.628 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613037773.753887_555.1","last_update":"2021-02-11T10:02:53.753887Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:02:54.628 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T10:02:54.629 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:02:54.629 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:02:54.629 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:02:54.671 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:02:54.671 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:02:54.902 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037773.753887_555.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:02:53.753887Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:02:54.903 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T10:02:54.903 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T10:02:54.903 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:02:54.903 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:02:54.903 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:02:54.941 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:02:54.941 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:02:55.168 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037773.753887_555.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:02:53.753887Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:02:55.169 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T10:02:55.169 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:02:55.169 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:02:55.169 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:02:55.169 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:02:55.210 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:02:55.211 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:02:55.442 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037773.753887_555.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:02:53.753887Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:02:55.442 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:02:55.442 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:02:55.443 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:02:55.443 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-62 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:02:55.532 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:02:55.532 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:02:55.775 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:02:55.769+0000 7f279a4e2080 1 bucket sync caught up with source: 2021-02-11T10:02:55.775 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [00000000001.46.5, , 00000000001.51.5, , 00000000001.35.5, , , 00000000001.53.5, , , ] 2021-02-11T10:02:55.775 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [00000000001.46.5, , 00000000001.51.5, , 00000000001.35.5, , , 00000000001.53.5, , , ] 2021-02-11T10:02:55.775 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:02:55.769+0000 7f279a4e2080 0 bucket checkpoint complete 2021-02-11T10:02:55.812 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:02:55.812 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:02:55.812 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-62 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:02:55.859 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:02:55.859 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:02:56.073 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:02:56.070+0000 7f61e8433080 1 bucket sync caught up with source: 2021-02-11T10:02:56.073 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [00000000001.46.5, , 00000000001.51.5, , 00000000001.35.5, , , 00000000001.53.5, , , ] 2021-02-11T10:02:56.073 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [00000000001.46.5, , 00000000001.51.5, , 00000000001.35.5, , , 00000000001.53.5, , , ] 2021-02-11T10:02:56.073 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:02:56.070+0000 7f61e8433080 0 bucket checkpoint complete 2021-02-11T10:02:56.102 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:02:56.102 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:02:56.102 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-62 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:02:56.150 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:02:56.150 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:02:56.371 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:02:56.365+0000 7fc1d21a2080 1 bucket sync caught up with source: 2021-02-11T10:02:56.371 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [00000000001.46.5, , 00000000001.51.5, , 00000000001.35.5, , , 00000000001.53.5, , , ] 2021-02-11T10:02:56.371 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [00000000001.46.5, , 00000000001.51.5, , 00000000001.35.5, , , 00000000001.53.5, , , ] 2021-02-11T10:02:56.371 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:02:56.365+0000 7fc1d21a2080 0 bucket checkpoint complete 2021-02-11T10:02:56.404 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-62', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:02:56.404 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-62', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:02:56.405 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-62 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:02:56.452 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:02:56.452 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:02:56.677 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:02:56.674+0000 7f1a15765080 1 bucket sync caught up with source: 2021-02-11T10:02:56.677 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [00000000001.52.5, , 00000000001.64.5, , 00000000001.35.5, , , 00000000001.68.5, , , ] 2021-02-11T10:02:56.677 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [00000000001.52.5, , 00000000001.64.5, , 00000000001.35.5, , , 00000000001.68.5, , , ] 2021-02-11T10:02:56.677 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:02:56.674+0000 7f1a15765080 0 bucket checkpoint complete 2021-02-11T10:02:56.712 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-62', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:02:56.712 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-62', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:02:56.712 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-62 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:02:56.760 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:02:56.760 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:02:56.983 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:02:56.982+0000 7fcc0adf1080 1 bucket sync caught up with source: 2021-02-11T10:02:56.983 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [00000000001.52.5, , 00000000001.64.5, , 00000000001.35.5, , , 00000000001.68.5, , , ] 2021-02-11T10:02:56.983 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [00000000001.52.5, , 00000000001.64.5, , 00000000001.35.5, , , 00000000001.68.5, , , ] 2021-02-11T10:02:56.983 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:02:56.982+0000 7fcc0adf1080 0 bucket checkpoint complete 2021-02-11T10:02:57.017 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-62', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:02:57.017 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-62', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:02:57.017 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-62 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:02:57.060 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:02:57.060 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:02:57.278 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:02:57.274+0000 7fd191925080 1 bucket sync caught up with source: 2021-02-11T10:02:57.279 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [00000000001.44.5, , 00000000001.58.5, , 00000000001.35.5, , , 00000000001.60.5, , , ] 2021-02-11T10:02:57.279 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [00000000001.44.5, , 00000000001.58.5, , 00000000001.35.5, , , 00000000001.60.5, , , ] 2021-02-11T10:02:57.279 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:02:57.274+0000 7fd191925080 0 bucket checkpoint complete 2021-02-11T10:02:57.305 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-62', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:02:57.305 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-62', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:02:57.305 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-62 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:02:57.353 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:02:57.353 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:02:57.566 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:02:57.561+0000 7fd34bcaa080 1 bucket sync caught up with source: 2021-02-11T10:02:57.567 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [00000000001.44.5, , 00000000001.58.5, , 00000000001.35.5, , , 00000000001.60.5, , , ] 2021-02-11T10:02:57.567 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [00000000001.44.5, , 00000000001.58.5, , 00000000001.35.5, , , 00000000001.60.5, , , ] 2021-02-11T10:02:57.567 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:02:57.561+0000 7fd34bcaa080 0 bucket checkpoint complete 2021-02-11T10:02:57.607 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-62 zones={test-zone2, test-zone1} 2021-02-11T10:02:57.618 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:02:57.623 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:02:57.624 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:02:57.624 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:02:57.624 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:02:57.624 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:02:57.630 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:02:57.630 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:02:57.630 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:02:57.630 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:02:57.641 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:02:57.695 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:02:57.807 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:02:57.914 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:02:57.968 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:02:58.079 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:02:58.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:02:58.239 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:02:58.353 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:02:58.457 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:02:58.511 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:02:58.620 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:02:58.726 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-62 zones={test-zone2, test-zone1} 2021-02-11T10:02:58.726 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-62 zones={test-zone3, test-zone1} 2021-02-11T10:02:58.735 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:02:58.739 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:02:58.739 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:02:58.739 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:02:58.739 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:02:58.739 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:02:58.743 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:02:58.744 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:02:58.744 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:02:58.744 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:02:58.752 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:02:58.803 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:02:58.910 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:02:59.014 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:02:59.067 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:02:59.175 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:02:59.318 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:02:59.372 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:02:59.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:02:59.654 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:02:59.716 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:02:59.829 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:02:59.937 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-62 zones={test-zone3, test-zone1} 2021-02-11T10:02:59.938 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-62 zones={test-zone3, test-zone2} 2021-02-11T10:02:59.948 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:02:59.953 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:02:59.953 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:02:59.953 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:02:59.953 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:02:59.953 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:03:00.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:03:00.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:03:00.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:03:00.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:03:00.058 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:03:00.113 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:03:00.224 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:03:00.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:03:00.389 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:03:00.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:03:00.604 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:03:00.660 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:03:00.771 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:03:00.877 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:03:00.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:03:01.043 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:03:01.145 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-62 zones={test-zone3, test-zone2} 2021-02-11T10:03:01.146 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:03:01.146 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:03:01.146 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-63 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:03:01.189 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:03:01.189 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:03:01.427 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:03:01.421+0000 7fd00fa2f080 1 bucket sync caught up with source: 2021-02-11T10:03:01.427 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [00000000001.37.5, , 00000000001.85.5, , 00000000001.48.5, , , 00000000001.55.5, , , ] 2021-02-11T10:03:01.427 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [00000000001.37.5, , 00000000001.85.5, , 00000000001.48.5, , , 00000000001.55.5, , , ] 2021-02-11T10:03:01.427 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:03:01.421+0000 7fd00fa2f080 0 bucket checkpoint complete 2021-02-11T10:03:01.463 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:03:01.463 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:03:01.463 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-63 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:03:01.510 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:03:01.510 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:03:01.738 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:03:01.734+0000 7f8a39974080 1 bucket sync caught up with source: 2021-02-11T10:03:01.738 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [00000000001.37.5, , 00000000001.85.5, , 00000000001.48.5, , , 00000000001.55.5, , , ] 2021-02-11T10:03:01.738 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [00000000001.37.5, , 00000000001.85.5, , 00000000001.48.5, , , 00000000001.55.5, , , ] 2021-02-11T10:03:01.739 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:03:01.734+0000 7f8a39974080 0 bucket checkpoint complete 2021-02-11T10:03:01.768 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:03:01.768 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:03:01.769 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-63 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:03:01.816 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:03:01.816 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:03:02.039 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:03:02.033+0000 7f56d9b41080 1 bucket sync caught up with source: 2021-02-11T10:03:02.039 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [00000000001.37.5, , 00000000001.85.5, , 00000000001.48.5, , , 00000000001.55.5, , , ] 2021-02-11T10:03:02.039 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [00000000001.37.5, , 00000000001.85.5, , 00000000001.48.5, , , 00000000001.55.5, , , ] 2021-02-11T10:03:02.039 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:03:02.033+0000 7f56d9b41080 0 bucket checkpoint complete 2021-02-11T10:03:02.073 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-63', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:03:02.073 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-63', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:03:02.073 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-63 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:03:02.121 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:03:02.122 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:03:02.346 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:03:02.342+0000 7fcfeb0f5080 1 bucket sync caught up with source: 2021-02-11T10:03:02.346 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [00000000001.37.5, , 00000000001.81.5, , 00000000001.50.5, , , 00000000001.66.5, , , ] 2021-02-11T10:03:02.346 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [00000000001.37.5, , 00000000001.81.5, , 00000000001.50.5, , , 00000000001.66.5, , , ] 2021-02-11T10:03:02.346 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:03:02.342+0000 7fcfeb0f5080 0 bucket checkpoint complete 2021-02-11T10:03:02.374 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-63', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:03:02.374 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-63', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:03:02.375 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-63 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:03:02.417 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:03:02.417 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:03:02.639 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:03:02.638+0000 7fdb32823080 1 bucket sync caught up with source: 2021-02-11T10:03:02.639 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [00000000001.37.5, , 00000000001.81.5, , 00000000001.50.5, , , 00000000001.66.5, , , ] 2021-02-11T10:03:02.640 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [00000000001.37.5, , 00000000001.81.5, , 00000000001.50.5, , , 00000000001.66.5, , , ] 2021-02-11T10:03:02.640 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:03:02.638+0000 7fdb32823080 0 bucket checkpoint complete 2021-02-11T10:03:02.673 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-63', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:03:02.673 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-63', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:03:02.673 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-63 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:03:02.762 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:03:02.762 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:03:02.980 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:03:02.978+0000 7fc34e109080 1 bucket sync caught up with source: 2021-02-11T10:03:02.980 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [00000000001.37.5, , 00000000001.76.5, , 00000000001.46.5, , , 00000000001.62.5, , , ] 2021-02-11T10:03:02.980 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [00000000001.37.5, , 00000000001.76.5, , 00000000001.46.5, , , 00000000001.62.5, , , ] 2021-02-11T10:03:02.980 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:03:02.978+0000 7fc34e109080 0 bucket checkpoint complete 2021-02-11T10:03:03.011 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-63', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:03:03.011 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-63', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:03:03.012 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-63 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:03:03.060 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:03:03.060 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:03:03.293 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:03:03.285+0000 7f4da8040080 1 bucket sync caught up with source: 2021-02-11T10:03:03.294 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [00000000001.37.5, , 00000000001.76.5, , 00000000001.46.5, , , 00000000001.62.5, , , ] 2021-02-11T10:03:03.294 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [00000000001.37.5, , 00000000001.76.5, , 00000000001.46.5, , , 00000000001.62.5, , , ] 2021-02-11T10:03:03.294 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:03:03.285+0000 7f4da8040080 0 bucket checkpoint complete 2021-02-11T10:03:03.327 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-63 zones={test-zone2, test-zone1} 2021-02-11T10:03:03.335 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:03:03.341 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:03:03.341 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:03:03.341 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:03:03.341 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:03:03.341 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:03:03.346 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:03:03.346 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:03:03.346 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:03:03.346 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:03:03.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:03:03.452 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:03:03.558 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:03:03.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:03:03.715 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:03:03.828 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:03:03.934 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:03:03.987 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:03:04.096 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:03:04.200 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:03:04.257 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:03:04.371 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:03:04.477 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-63 zones={test-zone2, test-zone1} 2021-02-11T10:03:04.477 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-63 zones={test-zone3, test-zone1} 2021-02-11T10:03:04.486 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:03:04.491 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:03:04.491 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:03:04.491 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:03:04.491 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:03:04.491 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:03:04.495 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:03:04.495 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:03:04.495 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:03:04.495 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:03:04.504 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:03:04.558 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:03:04.668 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:03:04.769 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:03:04.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:03:04.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:03:05.034 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:03:05.088 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:03:05.198 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:03:05.301 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:03:05.353 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:03:05.467 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:03:05.569 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-63 zones={test-zone3, test-zone1} 2021-02-11T10:03:05.569 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-63 zones={test-zone3, test-zone2} 2021-02-11T10:03:05.580 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:03:05.585 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:03:05.585 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:03:05.585 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:03:05.586 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:03:05.586 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:03:05.592 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:03:05.592 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:03:05.592 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:03:05.592 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:03:05.604 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:03:05.655 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:03:05.763 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:03:05.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:03:05.919 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:03:06.025 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:03:06.129 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:03:06.188 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:03:06.299 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:03:06.406 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:03:06.462 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:03:06.571 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:03:06.677 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-63 zones={test-zone3, test-zone2} 2021-02-11T10:03:06.678 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:03:06.678 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:03:06.678 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-64 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:03:06.720 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:03:06.720 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:03:06.957 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:03:06.949+0000 7f04a9ed9080 1 bucket sync caught up with source: 2021-02-11T10:03:06.957 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [00000000001.64.5, , 00000000001.34.5, , 00000000001.55.5, , , 00000000001.50.5, , , ] 2021-02-11T10:03:06.958 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [00000000001.64.5, , 00000000001.34.5, , 00000000001.55.5, , , 00000000001.50.5, , , ] 2021-02-11T10:03:06.958 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:03:06.949+0000 7f04a9ed9080 0 bucket checkpoint complete 2021-02-11T10:03:06.994 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:03:06.994 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:03:06.994 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-64 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:03:07.042 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:03:07.042 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:03:07.256 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:03:07.253+0000 7fd23755b080 1 bucket sync caught up with source: 2021-02-11T10:03:07.256 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [00000000001.64.5, , 00000000001.34.5, , 00000000001.55.5, , , 00000000001.50.5, , , ] 2021-02-11T10:03:07.257 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [00000000001.64.5, , 00000000001.34.5, , 00000000001.55.5, , , 00000000001.50.5, , , ] 2021-02-11T10:03:07.257 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:03:07.253+0000 7fd23755b080 0 bucket checkpoint complete 2021-02-11T10:03:07.284 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:03:07.284 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:03:07.285 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-64 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:03:07.337 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:03:07.337 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:03:07.543 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:03:07.537+0000 7fb949fbb080 1 waiting for incremental sync to catch up: 2021-02-11T10:03:07.543 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, , , , 00000000001.55.5, , , 00000000001.50.5, , , ] 2021-02-11T10:03:07.543 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [00000000001.64.5, , 00000000001.34.5, , 00000000001.55.5, , , 00000000001.50.5, , , ] 2021-02-11T10:03:12.547 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:03:12.541+0000 7fb949fbb080 1 waiting for incremental sync to catch up: 2021-02-11T10:03:12.547 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, , , , 00000000001.55.5, , , 00000000001.50.5, , , ] 2021-02-11T10:03:12.547 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [00000000001.64.5, , 00000000001.34.5, , 00000000001.55.5, , , 00000000001.50.5, , , ] 2021-02-11T10:03:17.551 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:03:17.544+0000 7fb949fbb080 1 bucket sync caught up with source: 2021-02-11T10:03:17.551 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [00000000001.64.5, , 00000000001.34.5, , 00000000001.55.5, , , 00000000001.50.5, , , ] 2021-02-11T10:03:17.551 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [00000000001.64.5, , 00000000001.34.5, , 00000000001.55.5, , , 00000000001.50.5, , , ] 2021-02-11T10:03:17.551 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:03:17.544+0000 7fb949fbb080 0 bucket checkpoint complete 2021-02-11T10:03:17.585 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-64', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:03:17.585 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-64', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:03:17.586 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-64 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:03:17.632 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:03:17.632 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:03:17.862 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:03:17.857+0000 7fa64a38e080 1 bucket sync caught up with source: 2021-02-11T10:03:17.862 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [00000000001.65.5, , 00000000001.36.5, , 00000000001.51.5, , , 00000000001.51.5, , , ] 2021-02-11T10:03:17.862 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [00000000001.65.5, , 00000000001.36.5, , 00000000001.51.5, , , 00000000001.51.5, , , ] 2021-02-11T10:03:17.862 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:03:17.857+0000 7fa64a38e080 0 bucket checkpoint complete 2021-02-11T10:03:17.898 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-64', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:03:17.899 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-64', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:03:17.899 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-64 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:03:17.943 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:03:17.943 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:03:18.173 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:03:18.169+0000 7fd11c746080 1 bucket sync caught up with source: 2021-02-11T10:03:18.173 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [00000000001.65.5, , 00000000001.36.5, , 00000000001.51.5, , , 00000000001.51.5, , , ] 2021-02-11T10:03:18.173 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [00000000001.65.5, , 00000000001.36.5, , 00000000001.51.5, , , 00000000001.51.5, , , ] 2021-02-11T10:03:18.173 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:03:18.169+0000 7fd11c746080 0 bucket checkpoint complete 2021-02-11T10:03:18.210 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-64', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:03:18.210 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-64', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:03:18.210 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-64 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:03:18.254 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:03:18.254 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:03:18.466 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:03:18.461+0000 7fa720b3a080 1 bucket sync caught up with source: 2021-02-11T10:03:18.466 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [00000000001.70.5, , 00000000001.36.5, , 00000000001.48.5, , , 00000000001.52.5, , , ] 2021-02-11T10:03:18.467 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [00000000001.70.5, , 00000000001.36.5, , 00000000001.48.5, , , 00000000001.52.5, , , ] 2021-02-11T10:03:18.467 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:03:18.461+0000 7fa720b3a080 0 bucket checkpoint complete 2021-02-11T10:03:18.496 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-64', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:03:18.497 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-64', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:03:18.497 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-64 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:03:18.546 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:03:18.546 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:03:18.780 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:03:18.772+0000 7f341aec6080 1 bucket sync caught up with source: 2021-02-11T10:03:18.781 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [00000000001.70.5, , 00000000001.36.5, , 00000000001.48.5, , , 00000000001.52.5, , , ] 2021-02-11T10:03:18.781 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [00000000001.70.5, , 00000000001.36.5, , 00000000001.48.5, , , 00000000001.52.5, , , ] 2021-02-11T10:03:18.781 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:03:18.772+0000 7f341aec6080 0 bucket checkpoint complete 2021-02-11T10:03:18.816 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-64 zones={test-zone2, test-zone1} 2021-02-11T10:03:18.827 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:03:18.875 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:03:18.875 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:03:18.875 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:03:18.875 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:03:18.875 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:03:18.882 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:03:18.882 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:03:18.883 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:03:18.883 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:03:18.942 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:03:18.996 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:03:19.108 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:03:19.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:03:19.272 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:03:19.382 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:03:19.487 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:03:19.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:03:19.651 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:03:19.759 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:03:19.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:03:19.920 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:03:20.027 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-64 zones={test-zone2, test-zone1} 2021-02-11T10:03:20.027 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-64 zones={test-zone3, test-zone1} 2021-02-11T10:03:20.035 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:03:20.040 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:03:20.040 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:03:20.040 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:03:20.040 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:03:20.040 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:03:20.045 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:03:20.045 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:03:20.045 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:03:20.045 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:03:20.054 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:03:20.107 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:03:20.219 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:03:20.322 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:03:20.376 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:03:20.487 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:03:20.589 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:03:20.643 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:03:20.753 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:03:20.858 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:03:20.912 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:03:21.022 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:03:21.126 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-64 zones={test-zone3, test-zone1} 2021-02-11T10:03:21.126 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-64 zones={test-zone3, test-zone2} 2021-02-11T10:03:21.138 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:03:21.143 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:03:21.143 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:03:21.143 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:03:21.144 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:03:21.144 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:03:21.191 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:03:21.191 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:03:21.191 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:03:21.191 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:03:21.247 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:03:21.300 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:03:21.408 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:03:21.515 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:03:21.571 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:03:21.684 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:03:21.792 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:03:21.848 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:03:21.964 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:03:22.070 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:03:22.124 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:03:22.236 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:03:22.343 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-64 zones={test-zone3, test-zone2} 2021-02-11T10:03:22.343 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'tkscyf-62', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:03:22.343 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'tkscyf-62', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:03:22.343 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket tkscyf-62 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T10:03:22.387 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:03:22.387 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:03:22.691 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'tkscyf-63', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:03:22.691 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'tkscyf-63', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:03:22.691 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket tkscyf-63 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T10:03:22.733 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:03:22.734 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:03:23.020 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'tkscyf-64', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:03:23.020 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'tkscyf-64', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:03:23.020 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket tkscyf-64 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T10:03:23.101 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:03:23.101 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:03:23.401 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:03:23.402 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:03:23.402 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:03:23.484 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:03:23.484 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:03:23.688 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 2021-02-11T10:03:23.727 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613037803.352031_572.1","last_update":"2021-02-11T10:03:23.352031Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:03:23.727 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T10:03:23.728 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:03:23.728 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:03:23.728 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:03:23.770 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:03:23.770 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:03:24.003 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037802.643240_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:03:22.643240Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:03:24.004 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613037803.352031_572.1 target=1_1613037802.643240_565.1 2021-02-11T10:03:29.004 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:03:29.005 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:03:29.005 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:03:29.044 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:03:29.044 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:03:29.297 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037802.643240_565.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:03:22.643240Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:03:29.298 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613037803.352031_572.1 target=1_1613037802.643240_565.1 2021-02-11T10:03:34.299 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:03:34.299 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:03:34.299 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:03:34.337 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:03:34.338 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:03:34.572 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037803.352031_572.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:03:23.352031Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:03:34.573 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T10:03:34.573 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T10:03:34.573 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:03:34.573 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:03:34.573 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:03:34.619 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:03:34.619 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:03:34.810 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037803.352031_572.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:03:23.352031Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:03:34.810 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T10:03:34.810 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:03:34.810 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:03:34.810 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:03:34.811 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:03:34.857 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:03:34.858 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:03:35.077 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037803.352031_572.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:03:23.352031Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:03:35.078 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:03:35.666 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'tkscyf-62', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:03:35.667 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'tkscyf-62', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:03:35.667 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket tkscyf-62 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T10:03:35.713 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:03:35.713 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:03:35.980 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'tkscyf-63', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:03:35.980 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'tkscyf-63', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:03:35.980 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket tkscyf-63 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T10:03:36.018 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:03:36.018 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:03:36.345 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'tkscyf-64', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:03:36.345 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'tkscyf-64', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:03:36.345 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket tkscyf-64 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T10:03:36.422 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:03:36.422 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:03:36.831 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:03:36.831 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:03:36.831 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-62 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:03:36.883 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:03:36.883 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:03:37.100 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:03:37.092+0000 7f9538ac7080 1 waiting for incremental sync to catch up: 2021-02-11T10:03:37.100 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [00000000006.56.1, 00000000004.43.1, 00000000006.68.1, 00000000004.72.1, 00000000006.45.1, 00000000004.76.1, 00000000004.44.1, 00000000006.69.1, , 00000000004.74.1, ] 2021-02-11T10:03:37.101 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [00000000006.56.1, 00000000004.43.1, 00000000006.68.1, 00000000004.72.1, 00000000006.45.1, 00000000004.76.1, 00000000004.44.1, 00000000006.69.1, 00000000004.36.1, 00000000004.74.1, 00000000004.47.1] 2021-02-11T10:03:42.105 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:03:42.096+0000 7f9538ac7080 1 bucket sync caught up with source: 2021-02-11T10:03:42.105 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [00000000006.56.1, 00000000004.43.1, 00000000006.68.1, 00000000004.72.1, 00000000006.45.1, 00000000004.76.1, 00000000004.44.1, 00000000006.69.1, 00000000004.36.1, 00000000004.74.1, 00000000004.47.1] 2021-02-11T10:03:42.105 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [00000000006.56.1, 00000000004.43.1, 00000000006.68.1, 00000000004.72.1, 00000000006.45.1, 00000000004.76.1, 00000000004.44.1, 00000000006.69.1, 00000000004.36.1, 00000000004.74.1, 00000000004.47.1] 2021-02-11T10:03:42.105 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:03:42.096+0000 7f9538ac7080 0 bucket checkpoint complete 2021-02-11T10:03:42.139 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:03:42.139 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:03:42.139 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-62 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:03:42.186 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:03:42.187 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:03:42.397 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:03:42.393+0000 7f547d98a080 1 waiting for incremental sync to catch up: 2021-02-11T10:03:42.397 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [00000000006.56.1, , 00000000006.68.1, 00000000004.72.1, 00000000006.45.1, , 00000000004.44.1, 00000000006.69.1, 00000000004.36.1, , 00000000004.47.1] 2021-02-11T10:03:42.397 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [00000000006.56.1, 00000000004.43.1, 00000000006.68.1, 00000000004.72.1, 00000000006.45.1, 00000000004.76.1, 00000000004.44.1, 00000000006.69.1, 00000000004.36.1, 00000000004.74.1, 00000000004.47.1] 2021-02-11T10:03:47.401 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:03:47.397+0000 7f547d98a080 1 waiting for incremental sync to catch up: 2021-02-11T10:03:47.401 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [00000000006.56.1, , 00000000006.68.1, 00000000004.72.1, 00000000006.45.1, , 00000000004.44.1, 00000000006.69.1, 00000000004.36.1, , 00000000004.47.1] 2021-02-11T10:03:47.401 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [00000000006.56.1, 00000000004.43.1, 00000000006.68.1, 00000000004.72.1, 00000000006.45.1, 00000000004.76.1, 00000000004.44.1, 00000000006.69.1, 00000000004.36.1, 00000000004.74.1, 00000000004.47.1] 2021-02-11T10:03:52.404 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:03:52.401+0000 7f547d98a080 1 waiting for incremental sync to catch up: 2021-02-11T10:03:57.997 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [00000000006.56.1, , 00000000006.68.1, 00000000004.72.1, 00000000006.45.1, , 00000000004.44.1, 00000000006.69.1, 00000000004.36.1, , 00000000004.47.1] 2021-02-11T10:03:57.997 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [00000000006.56.1, 00000000004.43.1, 00000000006.68.1, 00000000004.72.1, 00000000006.45.1, 00000000004.76.1, 00000000004.44.1, 00000000006.69.1, 00000000004.36.1, 00000000004.74.1, 00000000004.47.1] 2021-02-11T10:03:57.998 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:03:57.405+0000 7f547d98a080 1 bucket sync caught up with source: 2021-02-11T10:03:57.998 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [00000000006.56.1, 00000000004.43.1, 00000000006.68.1, 00000000004.72.1, 00000000006.45.1, 00000000004.76.1, 00000000004.44.1, 00000000006.69.1, 00000000004.36.1, 00000000004.74.1, 00000000004.47.1] 2021-02-11T10:03:57.998 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [00000000006.56.1, 00000000004.43.1, 00000000006.68.1, 00000000004.72.1, 00000000006.45.1, 00000000004.76.1, 00000000004.44.1, 00000000006.69.1, 00000000004.36.1, 00000000004.74.1, 00000000004.47.1] 2021-02-11T10:03:57.998 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:03:57.405+0000 7f547d98a080 0 bucket checkpoint complete 2021-02-11T10:03:57.999 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:03:57.999 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:03:57.999 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-62 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:03:58.047 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:03:58.047 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:03:58.267 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:03:58.260+0000 7f6251749080 1 bucket sync caught up with source: 2021-02-11T10:03:58.267 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [00000000006.56.1, 00000000004.43.1, 00000000006.68.1, 00000000004.72.1, 00000000006.45.1, 00000000004.76.1, 00000000004.44.1, 00000000006.69.1, 00000000004.36.1, 00000000004.74.1, 00000000004.47.1] 2021-02-11T10:03:58.267 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [00000000006.56.1, 00000000004.43.1, 00000000006.68.1, 00000000004.72.1, 00000000006.45.1, 00000000004.76.1, 00000000004.44.1, 00000000006.69.1, 00000000004.36.1, 00000000004.74.1, 00000000004.47.1] 2021-02-11T10:03:58.267 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:03:58.260+0000 7f6251749080 0 bucket checkpoint complete 2021-02-11T10:03:58.308 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-62', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:03:58.308 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-62', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:03:58.308 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-62 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:03:58.356 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:03:58.357 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:03:58.586 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:03:58.581+0000 7f5222711080 1 bucket sync caught up with source: 2021-02-11T10:03:58.586 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [00000000004.64.5, 00000000002.42.1, 00000000004.84.5, 00000000002.70.1, 00000000004.49.5, 00000000002.74.1, 00000000002.44.1, 00000000005.88.5, 00000000002.32.1, 00000000002.75.1, 00000000002.47.1] 2021-02-11T10:03:58.586 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [00000000004.64.5, 00000000002.42.1, 00000000004.84.5, 00000000002.70.1, 00000000004.49.5, 00000000002.74.1, 00000000002.44.1, 00000000005.88.5, 00000000002.32.1, 00000000002.75.1, 00000000002.47.1] 2021-02-11T10:03:58.586 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:03:58.581+0000 7f5222711080 0 bucket checkpoint complete 2021-02-11T10:03:58.621 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-62', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:03:58.622 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-62', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:03:58.622 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-62 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:03:58.668 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:03:58.668 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:03:58.887 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:03:58.885+0000 7fa20c84a080 1 bucket sync caught up with source: 2021-02-11T10:03:58.887 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [00000000004.64.5, 00000000002.42.1, 00000000004.84.5, 00000000002.70.1, 00000000004.49.5, 00000000002.74.1, 00000000002.44.1, 00000000005.88.5, 00000000002.32.1, 00000000002.75.1, 00000000002.47.1] 2021-02-11T10:03:58.887 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [00000000004.64.5, 00000000002.42.1, 00000000004.84.5, 00000000002.70.1, 00000000004.49.5, 00000000002.74.1, 00000000002.44.1, 00000000005.88.5, 00000000002.32.1, 00000000002.75.1, 00000000002.47.1] 2021-02-11T10:03:58.887 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:03:58.885+0000 7fa20c84a080 0 bucket checkpoint complete 2021-02-11T10:03:58.922 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-62', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:03:58.922 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-62', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:03:58.922 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-62 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:03:58.965 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:03:58.965 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:03:59.189 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:03:59.185+0000 7fbbd0d08080 1 bucket sync caught up with source: 2021-02-11T10:03:59.189 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [00000000004.58.5, 00000000002.45.1, 00000000004.80.5, 00000000002.79.1, 00000000004.49.5, 00000000002.97.1, 00000000002.46.1, 00000000004.79.5, 00000000002.38.1, 00000000002.71.1, 00000000002.44.1] 2021-02-11T10:03:59.190 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [00000000004.58.5, 00000000002.45.1, 00000000004.80.5, 00000000002.79.1, 00000000004.49.5, 00000000002.97.1, 00000000002.46.1, 00000000004.79.5, 00000000002.38.1, 00000000002.71.1, 00000000002.44.1] 2021-02-11T10:03:59.190 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:03:59.185+0000 7fbbd0d08080 0 bucket checkpoint complete 2021-02-11T10:03:59.219 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-62', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:03:59.219 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-62', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:03:59.219 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-62 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:03:59.266 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:03:59.266 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:03:59.501 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:03:59.492+0000 7f49a5f88080 1 bucket sync caught up with source: 2021-02-11T10:03:59.501 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [00000000004.58.5, 00000000002.45.1, 00000000004.80.5, 00000000002.79.1, 00000000004.49.5, 00000000002.97.1, 00000000002.46.1, 00000000004.79.5, 00000000002.38.1, 00000000002.71.1, 00000000002.44.1] 2021-02-11T10:03:59.501 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [00000000004.58.5, 00000000002.45.1, 00000000004.80.5, 00000000002.79.1, 00000000004.49.5, 00000000002.97.1, 00000000002.46.1, 00000000004.79.5, 00000000002.38.1, 00000000002.71.1, 00000000002.44.1] 2021-02-11T10:03:59.501 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:03:59.492+0000 7f49a5f88080 0 bucket checkpoint complete 2021-02-11T10:03:59.539 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-62 zones={test-zone2, test-zone1} 2021-02-11T10:03:59.549 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:03:59.556 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:03:59.556 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:03:59.556 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:03:59.556 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:03:59.556 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T10:03:59.556 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T10:03:59.556 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T10:03:59.556 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T10:03:59.556 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:03:59.563 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:03:59.563 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:03:59.563 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:03:59.563 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:03:59.563 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T10:03:59.563 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T10:03:59.563 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T10:03:59.563 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T10:03:59.575 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:03:59.627 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:03:59.780 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:03:59.886 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:03:59.939 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:04:00.051 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:04:00.154 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:04:00.208 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:04:00.319 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:04:00.425 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:04:00.479 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:04:00.587 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:04:00.689 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:04:00.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:04:00.857 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:04:00.985 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:04:01.042 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:04:01.157 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:04:01.266 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:04:01.318 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:04:01.423 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:04:01.529 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:04:01.583 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:04:01.695 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:04:01.801 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-62 zones={test-zone2, test-zone1} 2021-02-11T10:04:01.802 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-62 zones={test-zone3, test-zone1} 2021-02-11T10:04:01.811 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:04:01.815 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:04:01.816 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:04:01.816 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:04:01.816 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:04:01.816 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T10:04:01.816 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T10:04:01.816 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T10:04:01.816 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T10:04:01.816 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:04:01.820 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:04:01.821 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:04:01.821 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:04:01.821 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:04:01.821 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T10:04:01.821 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T10:04:01.821 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T10:04:01.821 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T10:04:01.830 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:04:01.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:04:01.990 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:04:02.093 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:04:02.147 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:04:02.255 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:04:02.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:04:02.410 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:04:02.519 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:04:02.625 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:04:02.679 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:04:02.786 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:04:02.889 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:04:02.944 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:04:03.058 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:04:03.161 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:04:03.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:04:03.330 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:04:03.433 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:04:03.487 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:04:03.599 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:04:03.700 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:04:03.756 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:04:03.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:04:03.969 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-62 zones={test-zone3, test-zone1} 2021-02-11T10:04:03.969 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-62 zones={test-zone3, test-zone2} 2021-02-11T10:04:03.979 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:04:03.985 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:04:03.985 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:04:03.985 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:04:03.985 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:04:03.985 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T10:04:03.985 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T10:04:03.985 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T10:04:03.985 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T10:04:03.985 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:04:03.992 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:04:03.992 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:04:03.992 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:04:03.992 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:04:03.992 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T10:04:03.992 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T10:04:03.992 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T10:04:03.992 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T10:04:04.003 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:04:04.055 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:04:04.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:04:04.266 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:04:04.324 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:04:04.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:04:04.536 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:04:04.590 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:04:04.699 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:04:04.801 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:04:04.855 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:04:04.975 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:04:05.077 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:04:05.131 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:04:05.242 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:04:05.345 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:04:05.397 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:04:05.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:04:05.610 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:04:05.662 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:04:05.771 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:04:05.873 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:04:05.927 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:04:06.039 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:04:06.147 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-62 zones={test-zone3, test-zone2} 2021-02-11T10:04:06.147 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:06.147 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:06.147 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-63 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:04:06.190 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:06.190 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:06.412 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:04:06.404+0000 7f4cf0cd3080 1 bucket sync caught up with source: 2021-02-11T10:04:06.412 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [00000000006.49.5, 00000000004.31.1, 00000000006.91.5, 00000000004.28.1, 00000000006.60.5, 00000000004.62.1, 00000000004.33.1, 00000000006.73.5, 00000000004.58.1, 00000000004.63.1, 00000000004.76.1] 2021-02-11T10:04:06.413 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [00000000006.49.5, 00000000004.31.1, 00000000006.91.5, 00000000004.28.1, 00000000006.60.5, 00000000004.62.1, 00000000004.33.1, 00000000006.73.5, 00000000004.58.1, 00000000004.63.1, 00000000004.76.1] 2021-02-11T10:04:06.413 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:04:06.404+0000 7f4cf0cd3080 0 bucket checkpoint complete 2021-02-11T10:04:06.446 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:06.447 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:06.447 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-63 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:04:06.493 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:06.493 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:06.720 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:04:06.717+0000 7fa70d730080 1 bucket sync caught up with source: 2021-02-11T10:04:06.720 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [00000000006.49.5, 00000000004.31.1, 00000000006.91.5, 00000000004.28.1, 00000000006.60.5, 00000000004.62.1, 00000000004.33.1, 00000000006.73.5, 00000000004.58.1, 00000000004.63.1, 00000000004.76.1] 2021-02-11T10:04:06.720 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [00000000006.49.5, 00000000004.31.1, 00000000006.91.5, 00000000004.28.1, 00000000006.60.5, 00000000004.62.1, 00000000004.33.1, 00000000006.73.5, 00000000004.58.1, 00000000004.63.1, 00000000004.76.1] 2021-02-11T10:04:06.720 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:04:06.717+0000 7fa70d730080 0 bucket checkpoint complete 2021-02-11T10:04:06.749 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:06.749 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:06.749 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-63 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:04:06.797 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:06.797 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:07.045 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:04:07.036+0000 7f2c83faa080 1 bucket sync caught up with source: 2021-02-11T10:04:07.045 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [00000000006.49.5, 00000000004.31.1, 00000000006.91.5, 00000000004.28.1, 00000000006.60.5, 00000000004.62.1, 00000000004.33.1, 00000000006.73.5, 00000000004.58.1, 00000000004.63.1, 00000000004.76.1] 2021-02-11T10:04:07.045 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [00000000006.49.5, 00000000004.31.1, 00000000006.91.5, 00000000004.28.1, 00000000006.60.5, 00000000004.62.1, 00000000004.33.1, 00000000006.73.5, 00000000004.58.1, 00000000004.63.1, 00000000004.76.1] 2021-02-11T10:04:07.045 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:04:07.036+0000 7f2c83faa080 0 bucket checkpoint complete 2021-02-11T10:04:07.079 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-63', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:07.079 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-63', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:07.079 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-63 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:04:07.127 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:07.127 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:07.348 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:04:07.345+0000 7fee1619e080 1 bucket sync caught up with source: 2021-02-11T10:04:07.348 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [00000000004.45.1, 00000000002.32.1, 00000000004.86.1, 00000000002.33.1, 00000000004.60.1, 00000000002.77.1, 00000000002.39.1, 00000000004.79.1, 00000000002.58.1, 00000000002.80.1, 00000000002.77.1] 2021-02-11T10:04:07.348 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [00000000004.45.1, 00000000002.32.1, 00000000004.86.1, 00000000002.33.1, 00000000004.60.1, 00000000002.77.1, 00000000002.39.1, 00000000004.79.1, 00000000002.58.1, 00000000002.80.1, 00000000002.77.1] 2021-02-11T10:04:07.348 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:04:07.345+0000 7fee1619e080 0 bucket checkpoint complete 2021-02-11T10:04:07.380 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-63', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:07.380 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-63', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:07.380 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-63 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:04:07.426 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:07.426 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:07.651 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:04:07.649+0000 7f0604ec7080 1 bucket sync caught up with source: 2021-02-11T10:04:07.651 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [00000000004.45.1, 00000000002.32.1, 00000000004.86.1, 00000000002.33.1, 00000000004.60.1, 00000000002.77.1, 00000000002.39.1, 00000000004.79.1, 00000000002.58.1, 00000000002.80.1, 00000000002.77.1] 2021-02-11T10:04:07.651 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [00000000004.45.1, 00000000002.32.1, 00000000004.86.1, 00000000002.33.1, 00000000004.60.1, 00000000002.77.1, 00000000002.39.1, 00000000004.79.1, 00000000002.58.1, 00000000002.80.1, 00000000002.77.1] 2021-02-11T10:04:07.652 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:04:07.649+0000 7f0604ec7080 0 bucket checkpoint complete 2021-02-11T10:04:07.684 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-63', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:07.685 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-63', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:07.685 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-63 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:04:07.773 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:07.773 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:07.995 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:04:07.993+0000 7f417fff9080 1 bucket sync caught up with source: 2021-02-11T10:04:07.996 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [00000000004.47.5, 00000000002.23.1, 00000000004.82.5, 00000000002.31.1, 00000000004.56.5, 00000000002.59.1, 00000000002.34.1, 00000000004.76.5, 00000000002.62.1, 00000000002.62.1, 00000000002.73.1] 2021-02-11T10:04:07.996 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [00000000004.47.5, 00000000002.23.1, 00000000004.82.5, 00000000002.31.1, 00000000004.56.5, 00000000002.59.1, 00000000002.34.1, 00000000004.76.5, 00000000002.62.1, 00000000002.62.1, 00000000002.73.1] 2021-02-11T10:04:07.996 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:04:07.993+0000 7f417fff9080 0 bucket checkpoint complete 2021-02-11T10:04:08.025 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-63', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:08.025 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-63', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:08.026 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-63 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:04:08.073 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:08.073 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:08.316 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:04:08.308+0000 7f483c7c7080 1 bucket sync caught up with source: 2021-02-11T10:04:08.316 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [00000000004.47.5, 00000000002.23.1, 00000000004.82.5, 00000000002.31.1, 00000000004.56.5, 00000000002.59.1, 00000000002.34.1, 00000000004.76.5, 00000000002.62.1, 00000000002.62.1, 00000000002.73.1] 2021-02-11T10:04:08.316 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [00000000004.47.5, 00000000002.23.1, 00000000004.82.5, 00000000002.31.1, 00000000004.56.5, 00000000002.59.1, 00000000002.34.1, 00000000004.76.5, 00000000002.62.1, 00000000002.62.1, 00000000002.73.1] 2021-02-11T10:04:08.316 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:04:08.308+0000 7f483c7c7080 0 bucket checkpoint complete 2021-02-11T10:04:08.352 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-63 zones={test-zone2, test-zone1} 2021-02-11T10:04:08.363 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:04:08.371 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:04:08.371 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:04:08.371 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:04:08.371 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:04:08.372 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T10:04:08.372 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T10:04:08.372 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T10:04:08.372 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T10:04:08.372 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:04:08.378 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:04:08.378 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:04:08.378 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:04:08.378 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:04:08.378 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T10:04:08.378 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T10:04:08.378 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T10:04:08.378 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T10:04:08.387 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:04:08.439 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:04:08.552 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:04:08.658 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:04:08.715 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:04:08.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:04:08.925 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:04:08.979 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:04:09.094 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:04:09.196 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:04:09.251 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:04:09.356 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:04:09.461 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:04:09.515 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:04:09.627 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:04:09.738 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:04:09.791 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:04:09.904 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:04:10.010 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:04:10.064 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:04:10.176 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:04:10.286 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:04:10.344 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:04:10.456 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:04:10.562 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-63 zones={test-zone2, test-zone1} 2021-02-11T10:04:10.562 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-63 zones={test-zone3, test-zone1} 2021-02-11T10:04:10.572 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:04:10.577 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:04:10.577 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:04:10.577 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:04:10.577 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:04:10.577 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T10:04:10.577 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T10:04:10.577 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T10:04:10.577 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T10:04:10.577 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:04:10.583 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:04:10.583 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:04:10.583 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:04:10.583 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:04:10.583 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T10:04:10.583 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T10:04:10.583 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T10:04:10.583 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T10:04:10.592 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:04:10.648 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:04:10.760 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:04:10.861 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:04:10.915 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:04:11.027 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:04:11.129 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:04:11.183 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:04:11.296 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:04:11.401 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:04:11.456 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:04:11.567 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:04:11.670 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:04:11.724 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:04:11.835 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:04:11.938 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:04:11.991 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:04:12.098 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:04:12.200 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:04:12.255 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:04:12.366 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:04:12.468 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:04:12.524 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:04:12.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:04:12.742 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-63 zones={test-zone3, test-zone1} 2021-02-11T10:04:12.742 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-63 zones={test-zone3, test-zone2} 2021-02-11T10:04:12.754 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:04:12.758 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:04:12.758 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:04:12.758 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:04:12.759 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:04:12.759 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T10:04:12.759 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T10:04:12.759 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T10:04:12.759 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T10:04:12.759 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:04:12.766 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:04:12.766 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:04:12.766 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:04:12.766 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:04:12.766 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T10:04:12.766 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T10:04:12.766 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T10:04:12.766 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T10:04:12.778 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:04:12.831 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:04:12.939 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:04:13.041 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:04:13.096 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:04:13.212 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:04:13.323 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:04:13.374 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:04:13.479 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:04:13.581 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:04:13.636 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:04:13.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:04:13.857 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:04:13.911 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:04:14.024 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:04:14.129 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:04:14.184 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:04:14.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:04:14.402 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:04:14.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:04:14.567 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:04:14.675 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:04:14.728 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:04:14.839 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:04:14.947 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-63 zones={test-zone3, test-zone2} 2021-02-11T10:04:14.947 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:14.947 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:14.948 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-64 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:04:14.993 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:14.993 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:15.237 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:04:15.228+0000 7f95a4e6e080 1 bucket sync caught up with source: 2021-02-11T10:04:15.237 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [00000000006.78.5, 00000000004.74.1, 00000000006.48.5, 00000000004.75.1, 00000000006.61.5, 00000000004.76.1, 00000000004.57.1, 00000000006.60.5, 00000000004.76.1, 00000000004.38.1, 00000000004.77.1] 2021-02-11T10:04:15.237 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [00000000006.78.5, 00000000004.74.1, 00000000006.48.5, 00000000004.75.1, 00000000006.61.5, 00000000004.76.1, 00000000004.57.1, 00000000006.60.5, 00000000004.76.1, 00000000004.38.1, 00000000004.77.1] 2021-02-11T10:04:15.237 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:04:15.228+0000 7f95a4e6e080 0 bucket checkpoint complete 2021-02-11T10:04:15.274 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:15.275 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:15.275 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-64 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:04:15.326 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:15.327 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:15.538 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:04:15.532+0000 7f499aac2080 1 bucket sync caught up with source: 2021-02-11T10:04:15.538 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [00000000006.78.5, 00000000004.74.1, 00000000006.48.5, 00000000004.75.1, 00000000006.61.5, 00000000004.76.1, 00000000004.57.1, 00000000006.60.5, 00000000004.76.1, 00000000004.38.1, 00000000004.77.1] 2021-02-11T10:04:15.538 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [00000000006.78.5, 00000000004.74.1, 00000000006.48.5, 00000000004.75.1, 00000000006.61.5, 00000000004.76.1, 00000000004.57.1, 00000000006.60.5, 00000000004.76.1, 00000000004.38.1, 00000000004.77.1] 2021-02-11T10:04:15.538 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:04:15.532+0000 7f499aac2080 0 bucket checkpoint complete 2021-02-11T10:04:15.567 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:15.567 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:15.567 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-64 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:04:15.615 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:15.615 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:15.837 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:04:15.828+0000 7f17d6960080 1 bucket sync caught up with source: 2021-02-11T10:04:15.838 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [00000000006.78.5, 00000000004.74.1, 00000000006.48.5, 00000000004.75.1, 00000000006.61.5, 00000000004.76.1, 00000000004.57.1, 00000000006.60.5, 00000000004.76.1, 00000000004.38.1, 00000000004.77.1] 2021-02-11T10:04:15.838 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [00000000006.78.5, 00000000004.74.1, 00000000006.48.5, 00000000004.75.1, 00000000006.61.5, 00000000004.76.1, 00000000004.57.1, 00000000006.60.5, 00000000004.76.1, 00000000004.38.1, 00000000004.77.1] 2021-02-11T10:04:15.838 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:04:15.828+0000 7f17d6960080 0 bucket checkpoint complete 2021-02-11T10:04:15.867 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-64', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:15.868 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-64', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:15.868 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-64 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:04:15.916 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:15.917 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:16.139 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:04:16.136+0000 7f41861bb080 1 bucket sync caught up with source: 2021-02-11T10:04:16.139 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [00000000004.76.1, 00000000002.66.1, 00000000004.49.1, 00000000002.71.1, 00000000004.56.1, 00000000002.77.1, 00000000002.54.1, 00000000004.59.5, 00000000002.72.1, 00000000002.34.1, 00000000002.73.1] 2021-02-11T10:04:16.139 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [00000000004.76.1, 00000000002.66.1, 00000000004.49.1, 00000000002.71.1, 00000000004.56.1, 00000000002.77.1, 00000000002.54.1, 00000000004.59.5, 00000000002.72.1, 00000000002.34.1, 00000000002.73.1] 2021-02-11T10:04:16.139 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:04:16.136+0000 7f41861bb080 0 bucket checkpoint complete 2021-02-11T10:04:16.171 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-64', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:16.172 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-64', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:16.172 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-64 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:04:16.217 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:16.217 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:16.442 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:04:16.436+0000 7f49d0256080 1 bucket sync caught up with source: 2021-02-11T10:04:16.442 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [00000000004.76.1, 00000000002.66.1, 00000000004.49.1, 00000000002.71.1, 00000000004.56.1, 00000000002.77.1, 00000000002.54.1, 00000000004.59.5, 00000000002.72.1, 00000000002.34.1, 00000000002.73.1] 2021-02-11T10:04:16.442 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [00000000004.76.1, 00000000002.66.1, 00000000004.49.1, 00000000002.71.1, 00000000004.56.1, 00000000002.77.1, 00000000002.54.1, 00000000004.59.5, 00000000002.72.1, 00000000002.34.1, 00000000002.73.1] 2021-02-11T10:04:16.442 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:04:16.436+0000 7f49d0256080 0 bucket checkpoint complete 2021-02-11T10:04:16.474 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-64', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:16.474 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-64', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:16.474 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-64 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:04:16.562 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:16.562 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:16.788 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:04:16.784+0000 7efee9c88080 1 bucket sync caught up with source: 2021-02-11T10:04:16.788 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [00000000004.80.1, 00000000002.68.1, 00000000004.47.1, 00000000002.73.1, 00000000004.53.1, 00000000002.82.1, 00000000002.59.1, 00000000004.60.1, 00000000002.74.1, 00000000002.40.1, 00000000002.75.1] 2021-02-11T10:04:16.788 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [00000000004.80.1, 00000000002.68.1, 00000000004.47.1, 00000000002.73.1, 00000000004.53.1, 00000000002.82.1, 00000000002.59.1, 00000000004.60.1, 00000000002.74.1, 00000000002.40.1, 00000000002.75.1] 2021-02-11T10:04:16.788 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:04:16.784+0000 7efee9c88080 0 bucket checkpoint complete 2021-02-11T10:04:16.817 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-64', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:16.817 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-64', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:16.817 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-64 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:04:16.866 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:16.866 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:17.106 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:04:17.100+0000 7ff67f04b080 1 bucket sync caught up with source: 2021-02-11T10:04:17.107 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [00000000004.80.1, 00000000002.68.1, 00000000004.47.1, 00000000002.73.1, 00000000004.53.1, 00000000002.82.1, 00000000002.59.1, 00000000004.60.1, 00000000002.74.1, 00000000002.40.1, 00000000002.75.1] 2021-02-11T10:04:17.107 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [00000000004.80.1, 00000000002.68.1, 00000000004.47.1, 00000000002.73.1, 00000000004.53.1, 00000000002.82.1, 00000000002.59.1, 00000000004.60.1, 00000000002.74.1, 00000000002.40.1, 00000000002.75.1] 2021-02-11T10:04:17.107 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:04:17.100+0000 7ff67f04b080 0 bucket checkpoint complete 2021-02-11T10:04:17.141 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-64 zones={test-zone2, test-zone1} 2021-02-11T10:04:17.153 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:04:17.160 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:04:17.160 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:04:17.160 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:04:17.160 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:04:17.160 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T10:04:17.161 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T10:04:17.161 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T10:04:17.161 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T10:04:17.161 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:04:17.167 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:04:17.167 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:04:17.167 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:04:17.167 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:04:17.167 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T10:04:17.167 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T10:04:17.167 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T10:04:17.167 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T10:04:17.181 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:04:17.236 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:04:17.345 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:04:17.451 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:04:17.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:04:17.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:04:17.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:04:17.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:04:17.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:04:17.991 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:04:18.048 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:04:18.161 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:04:18.271 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:04:18.324 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:04:18.427 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:04:18.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:04:18.588 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:04:18.705 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:04:18.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:04:18.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:04:18.981 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:04:19.086 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:04:19.140 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:04:19.253 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:04:19.360 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-64 zones={test-zone2, test-zone1} 2021-02-11T10:04:19.360 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-64 zones={test-zone3, test-zone1} 2021-02-11T10:04:19.369 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:04:19.375 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:04:19.375 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:04:19.375 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:04:19.375 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:04:19.375 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T10:04:19.375 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T10:04:19.375 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T10:04:19.375 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T10:04:19.376 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:04:19.380 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:04:19.380 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:04:19.380 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:04:19.381 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:04:19.381 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T10:04:19.381 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T10:04:19.381 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T10:04:19.381 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T10:04:19.391 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:04:19.442 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:04:19.552 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:04:19.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:04:19.708 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:04:19.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:04:19.921 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:04:19.975 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:04:20.088 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:04:20.194 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:04:20.247 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:04:20.358 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:04:20.462 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:04:20.515 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:04:20.631 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:04:20.734 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:04:20.787 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:04:20.900 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:04:21.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:04:21.059 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:04:21.167 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:04:21.270 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:04:21.321 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:04:21.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:04:21.579 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-64 zones={test-zone3, test-zone1} 2021-02-11T10:04:21.579 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-64 zones={test-zone3, test-zone2} 2021-02-11T10:04:21.590 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:04:21.598 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:04:21.598 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:04:21.598 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:04:21.598 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:04:21.598 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T10:04:21.598 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T10:04:21.598 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T10:04:21.599 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T10:04:21.599 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:04:21.606 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-02-11T10:04:21.606 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-02-11T10:04:21.606 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-02-11T10:04:21.606 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-02-11T10:04:21.606 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-02-11T10:04:21.606 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-02-11T10:04:21.606 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-02-11T10:04:21.606 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-02-11T10:04:21.622 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:04:21.674 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:04:21.789 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-02-11T10:04:21.895 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:04:21.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:04:22.061 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-02-11T10:04:22.167 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:04:22.220 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:04:22.333 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-02-11T10:04:22.548 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:04:22.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:04:22.725 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-02-11T10:04:22.830 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:04:22.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:04:22.993 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-02-11T10:04:23.103 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:04:23.163 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:04:23.281 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-02-11T10:04:23.383 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:04:23.435 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:04:23.549 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-02-11T10:04:23.655 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:04:23.708 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:04:23.820 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-02-11T10:04:23.927 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-64 zones={test-zone3, test-zone2} 2021-02-11T10:04:23.927 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:23.927 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:23.928 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:04:23.967 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:23.967 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:24.216 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613037816.810367_609.3","last_update":"2021-02-11T10:03:36.810367Z"},{"marker":"1_1613037818.016139_740.1","last_update":"2021-02-11T10:03:38.016139Z"},{"marker":"1_1613037818.088524_316.1","last_update":"2021-02-11T10:03:38.088524Z"},{"marker":"1_1613037818.022911_396.1","last_update":"2021-02-11T10:03:38.022911Z"}] 2021-02-11T10:04:24.217 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-02-11T10:04:24.217 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:24.217 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:24.217 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:04:24.260 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:24.260 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:24.519 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4703921989677143155},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037816.810367_609.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:03:36.810367Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037818.016139_740.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:03:38.016139Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613037818.088524_316.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:03:38.088524Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037818.022911_396.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:03:38.022911Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:04:24.519 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-02-11T10:04:24.519 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:24.519 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:24.520 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:04:24.559 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:24.559 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:24.803 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613037816.810367_609.3","last_update":"2021-02-11T10:03:36.810367Z"},{"marker":"1_1613037818.016139_740.1","last_update":"2021-02-11T10:03:38.016139Z"},{"marker":"1_1613037818.088524_316.1","last_update":"2021-02-11T10:03:38.088524Z"},{"marker":"1_1613037818.022911_396.1","last_update":"2021-02-11T10:03:38.022911Z"}] 2021-02-11T10:04:24.803 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-02-11T10:04:24.803 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:24.803 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:24.804 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:04:24.889 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:24.889 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:25.133 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":13312279424886386302},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037816.810367_609.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:03:36.810367Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037818.016139_740.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:03:38.016139Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613037818.088524_316.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:03:38.088524Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037818.022911_396.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:03:38.022911Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:04:25.134 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-02-11T10:04:25.134 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:25.134 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:25.134 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:04:25.214 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:25.214 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:25.451 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613037816.810367_609.3","last_update":"2021-02-11T10:03:36.810367Z"},{"marker":"1_1613037818.016139_740.1","last_update":"2021-02-11T10:03:38.016139Z"},{"marker":"1_1613037818.088524_316.1","last_update":"2021-02-11T10:03:38.088524Z"},{"marker":"1_1613037818.022911_396.1","last_update":"2021-02-11T10:03:38.022911Z"}] 2021-02-11T10:04:25.452 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-02-11T10:04:25.452 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:25.452 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:25.452 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:04:25.493 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:25.493 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:25.731 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2896161496309628191},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037816.810367_609.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:03:36.810367Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037818.016139_740.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:03:38.016139Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613037818.088524_316.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:03:38.088524Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037818.022911_396.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:03:38.022911Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:04:25.732 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-02-11T10:04:25.732 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:25.732 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:25.732 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:04:25.814 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:25.815 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:26.086 INFO:teuthology.orchestra.run.smithi098.stdout:[{"marker":"1_1613037819.376769_249.9","last_update":"2021-02-11T10:03:39.376769Z"},{"marker":"1_1613037819.376769_656.9","last_update":"2021-02-11T10:03:39.376769Z"},{"marker":"1_1613037819.376769_314.6","last_update":"2021-02-11T10:03:39.376769Z"},{"marker":"1_1613037819.376769_391.9","last_update":"2021-02-11T10:03:39.376769Z"}] 2021-02-11T10:04:26.086 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-02-11T10:04:26.086 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:26.086 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:26.086 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:04:26.129 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:26.129 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:26.378 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":15226599365335882107},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037819.376769_249.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:03:39.376769Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037819.376769_656.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:03:39.376769Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613037819.376769_314.6","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:03:39.376769Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037819.376769_391.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:03:39.376769Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:04:26.379 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-02-11T10:04:26.379 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:26.379 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:26.379 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:04:26.419 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:26.420 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:26.655 INFO:teuthology.orchestra.run.smithi098.stdout:[{"marker":"1_1613037819.376769_249.9","last_update":"2021-02-11T10:03:39.376769Z"},{"marker":"1_1613037819.376769_656.9","last_update":"2021-02-11T10:03:39.376769Z"},{"marker":"1_1613037819.376769_314.6","last_update":"2021-02-11T10:03:39.376769Z"},{"marker":"1_1613037819.376769_391.9","last_update":"2021-02-11T10:03:39.376769Z"}] 2021-02-11T10:04:29.083 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-02-11T10:04:29.083 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:29.083 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:29.083 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:04:29.127 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:29.127 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:29.373 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":10999913493642558030},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037819.376769_249.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:03:39.376769Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037819.376769_656.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:03:39.376769Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613037819.376769_314.6","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:03:39.376769Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037819.376769_391.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:03:39.376769Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:04:29.373 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-02-11T10:04:29.373 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:29.373 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:29.374 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:04:29.454 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:29.454 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:29.696 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613037819.246168_332.9","last_update":"2021-02-11T10:03:39.246168Z"},{"marker":"1_1613037819.246168_581.9","last_update":"2021-02-11T10:03:39.246168Z"},{"marker":"1_1613037819.246168_227.6","last_update":"2021-02-11T10:03:39.246168Z"},{"marker":"1_1613037819.246168_284.9","last_update":"2021-02-11T10:03:39.246168Z"}] 2021-02-11T10:04:29.696 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-02-11T10:04:29.697 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:29.697 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:29.697 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:04:29.779 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:29.779 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:30.026 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2753764877348348355},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037819.246168_332.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:03:39.246168Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037819.246168_581.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:03:39.246168Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613037819.246168_227.6","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:03:39.246168Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037819.246168_284.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:03:39.246168Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:04:30.026 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-02-11T10:04:30.026 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:30.026 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:30.027 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:04:30.104 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:30.104 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:30.347 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613037819.246168_332.9","last_update":"2021-02-11T10:03:39.246168Z"},{"marker":"1_1613037819.246168_581.9","last_update":"2021-02-11T10:03:39.246168Z"},{"marker":"1_1613037819.246168_227.6","last_update":"2021-02-11T10:03:39.246168Z"},{"marker":"1_1613037819.246168_284.9","last_update":"2021-02-11T10:03:39.246168Z"}] 2021-02-11T10:04:30.348 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-02-11T10:04:30.348 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:30.348 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:30.348 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:04:30.389 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:30.389 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:30.642 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":7143279782539285748},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037819.246168_332.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:03:39.246168Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037819.246168_581.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:03:39.246168Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613037819.246168_227.6","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:03:39.246168Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037819.246168_284.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:03:39.246168Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:04:30.643 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-02-11T10:04:30.643 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_sync_disable_enable ... ok 2021-02-11T10:04:30.644 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=tkscyf-65 2021-02-11T10:04:30.685 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=tkscyf-66 2021-02-11T10:04:30.767 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=tkscyf-67 2021-02-11T10:04:33.070 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:33.071 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:33.071 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-65 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:04:33.119 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:33.120 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:33.345 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:04:33.336+0000 7fa5046de080 1 bucket sync caught up with source: 2021-02-11T10:04:33.345 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, , , , , 00000000011.93.5, , , , , ] 2021-02-11T10:04:33.346 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, , , , , 00000000011.93.5, , , , , ] 2021-02-11T10:04:33.346 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:04:33.336+0000 7fa5046de080 0 bucket checkpoint complete 2021-02-11T10:04:33.386 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:33.386 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:33.386 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-65 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:04:33.432 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:33.432 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:33.656 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:04:33.652+0000 7f992d30c080 1 bucket sync caught up with source: 2021-02-11T10:04:33.656 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , 00000000011.93.5, , , , , ] 2021-02-11T10:04:33.657 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , 00000000011.93.5, , , , , ] 2021-02-11T10:04:33.657 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:04:33.652+0000 7f992d30c080 0 bucket checkpoint complete 2021-02-11T10:04:33.686 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:33.686 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:33.686 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-65 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:04:33.733 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:33.734 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:33.961 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:04:33.952+0000 7fccfcdf4080 1 bucket sync caught up with source: 2021-02-11T10:04:33.961 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, , , , , 00000000011.93.5, , , , , ] 2021-02-11T10:04:33.961 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, , , , , 00000000011.93.5, , , , , ] 2021-02-11T10:04:33.962 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:04:33.952+0000 7fccfcdf4080 0 bucket checkpoint complete 2021-02-11T10:04:33.995 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-65', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:33.995 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-65', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:33.995 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-65 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:04:34.042 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:34.043 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:34.282 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:04:34.276+0000 7ff1328ed080 1 bucket sync caught up with source: 2021-02-11T10:04:34.283 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , 00000000001.82.5, , , , , ] 2021-02-11T10:04:34.283 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , 00000000001.82.5, , , , , ] 2021-02-11T10:04:34.283 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:04:34.276+0000 7ff1328ed080 0 bucket checkpoint complete 2021-02-11T10:04:34.321 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-65', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:34.321 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-65', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:34.321 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-65 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:04:34.371 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:34.371 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:34.579 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:04:34.576+0000 7f813e53e080 1 bucket sync caught up with source: 2021-02-11T10:04:34.579 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , 00000000001.82.5, , , , , ] 2021-02-11T10:04:34.579 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , 00000000001.82.5, , , , , ] 2021-02-11T10:04:34.579 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:04:34.576+0000 7f813e53e080 0 bucket checkpoint complete 2021-02-11T10:04:34.613 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-65', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:34.614 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-65', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:34.614 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-65 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:04:34.657 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:34.657 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:34.887 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:04:34.884+0000 7f65a8bd9080 1 bucket sync caught up with source: 2021-02-11T10:04:34.887 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , 00000000001.87.5, , , , , ] 2021-02-11T10:04:34.887 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , 00000000001.87.5, , , , , ] 2021-02-11T10:04:34.887 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:04:34.884+0000 7f65a8bd9080 0 bucket checkpoint complete 2021-02-11T10:04:34.916 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-65', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:34.916 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-65', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:34.916 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-65 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:04:34.963 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:34.963 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:35.197 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:04:35.188+0000 7fa58b6d3080 1 bucket sync caught up with source: 2021-02-11T10:04:35.197 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, , , , , 00000000001.87.5, , , , , ] 2021-02-11T10:04:35.197 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, , , , , 00000000001.87.5, , , , , ] 2021-02-11T10:04:35.197 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:04:35.188+0000 7fa58b6d3080 0 bucket checkpoint complete 2021-02-11T10:04:35.233 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-65 zones={test-zone2, test-zone1} 2021-02-11T10:04:35.243 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:04:35.291 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2021-02-11T10:04:35.291 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:04:35.295 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2021-02-11T10:04:35.352 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2021-02-11T10:04:35.563 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2021-02-11T10:04:35.840 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2021-02-11T10:04:36.098 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-65 zones={test-zone2, test-zone1} 2021-02-11T10:04:36.098 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-65 zones={test-zone3, test-zone1} 2021-02-11T10:04:36.107 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:04:36.155 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2021-02-11T10:04:36.155 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:04:36.162 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2021-02-11T10:04:36.216 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2021-02-11T10:04:36.491 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2021-02-11T10:04:36.783 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2021-02-11T10:04:37.077 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-65 zones={test-zone3, test-zone1} 2021-02-11T10:04:37.077 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-65 zones={test-zone3, test-zone2} 2021-02-11T10:04:37.088 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:04:37.135 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2021-02-11T10:04:37.135 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:04:37.142 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2021-02-11T10:04:37.197 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2021-02-11T10:04:37.442 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2021-02-11T10:04:37.732 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2021-02-11T10:04:38.018 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-65 zones={test-zone3, test-zone2} 2021-02-11T10:04:38.018 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_multipart_object_sync ... ok 2021-02-11T10:04:38.019 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=tkscyf-68 2021-02-11T10:04:38.106 INFO:tasks.rgw.c2.client.1.smithi098.stdout:2021-02-11T10:04:38.100+0000 7fa80f7ee700 -1 data sync zone:dd308248 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T10:04:38.106 INFO:tasks.rgw.c2.client.0.smithi098.stdout:2021-02-11T10:04:38.100+0000 7f2bdaffd700 -1 data sync zone:dd308248 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T10:04:38.106 INFO:tasks.rgw.c1.client.1.smithi094.stdout:2021-02-11T10:04:38.100+0000 7f8fa6ff5700 -1 data sync zone:dd308248 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T10:04:38.146 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:38.147 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:38.147 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:04:38.182 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:38.182 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:38.358 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 2021-02-11T10:04:38.387 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613037878.056172_621.1","last_update":"2021-02-11T10:04:38.056172Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:04:38.387 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T10:04:38.387 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:38.387 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:38.388 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:04:38.423 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:38.424 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:38.648 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037878.056172_621.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:04:38.056172Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:04:38.648 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T10:04:38.648 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T10:04:38.648 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:38.648 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:38.649 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:04:38.691 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:38.691 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:38.904 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037878.056172_621.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:04:38.056172Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:04:38.905 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T10:04:38.905 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:04:38.905 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:38.905 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:38.905 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:04:38.946 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:38.946 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:39.191 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037878.056172_621.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:04:38.056172Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:04:39.192 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:04:39.192 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-68', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:39.192 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-68', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:39.192 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-68 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:04:39.280 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:39.280 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:39.515 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:04:39.508+0000 7f3d998ee080 1 bucket sync caught up with source: 2021-02-11T10:04:39.516 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, 00000000001.96.5, , , , 00000000001.67.5, , , , , ] 2021-02-11T10:04:39.516 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, 00000000001.96.5, , , , 00000000001.67.5, , , , , ] 2021-02-11T10:04:39.516 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:04:39.508+0000 7f3d998ee080 0 bucket checkpoint complete 2021-02-11T10:04:39.654 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_encrypted_object_sync ... ok 2021-02-11T10:04:39.655 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=tkscyf-69 2021-02-11T10:04:39.894 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:39.895 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:39.895 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:04:39.941 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:39.941 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:40.109 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 2021-02-11T10:04:40.146 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613037879.692653_626.1","last_update":"2021-02-11T10:04:39.692653Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:04:40.146 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T10:04:40.146 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:40.146 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:40.146 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:04:40.187 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:40.188 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:40.404 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037879.692653_626.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:04:39.692653Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:04:40.404 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T10:04:40.404 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T10:04:40.404 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:40.404 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:40.404 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:04:40.444 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:40.444 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:40.684 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037879.692653_626.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:04:39.692653Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:04:40.685 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T10:04:40.685 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:04:40.685 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:40.685 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:40.685 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:04:40.726 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:40.726 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:40.964 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037879.692653_626.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:04:39.692653Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:04:40.964 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:04:40.965 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-69', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:40.965 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-69', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:40.965 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-69 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:04:41.050 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:41.050 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:41.292 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:04:41.284+0000 7fa147af7080 1 bucket sync caught up with source: 2021-02-11T10:04:41.292 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, 00000000001.95.5, , , , 00000000001.74.5, , , 00000000001.65.5, , 00000000001.46.5] 2021-02-11T10:04:41.292 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, 00000000001.95.5, , , , 00000000001.74.5, , , 00000000001.65.5, , 00000000001.46.5] 2021-02-11T10:04:41.292 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:04:41.284+0000 7fa147af7080 0 bucket checkpoint complete 2021-02-11T10:04:41.329 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-69', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:41.329 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-69', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:41.329 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-69 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:04:41.379 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:41.379 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:41.599 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:04:41.596+0000 7f4639641080 1 bucket sync caught up with source: 2021-02-11T10:04:41.600 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000001.95.5, , , , 00000000001.74.5, , , 00000000001.65.5, , 00000000001.46.5] 2021-02-11T10:04:41.600 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000001.95.5, , , , 00000000001.74.5, , , 00000000001.65.5, , 00000000001.46.5] 2021-02-11T10:04:41.600 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:04:41.596+0000 7f4639641080 0 bucket checkpoint complete 2021-02-11T10:04:41.628 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-69', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:41.628 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-69', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:41.628 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-69 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:04:41.675 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:41.675 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:41.911 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:04:41.904+0000 7f790dc9d080 1 bucket sync caught up with source: 2021-02-11T10:04:41.911 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, 00000000001.95.5, , , , 00000000001.74.5, , , 00000000001.65.5, , 00000000001.46.5] 2021-02-11T10:04:41.911 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, 00000000001.95.5, , , , 00000000001.74.5, , , 00000000001.65.5, , 00000000001.46.5] 2021-02-11T10:04:41.911 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:04:41.904+0000 7f790dc9d080 0 bucket checkpoint complete 2021-02-11T10:04:41.945 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-69', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:41.945 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-69', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:41.945 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-69 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:04:41.993 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:41.993 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:42.227 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:04:42.224+0000 7fde050eb080 1 bucket sync caught up with source: 2021-02-11T10:04:42.227 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000001.91.5, , , , 00000000001.110.5, , , 00000000001.68.5, , 00000000001.40.5] 2021-02-11T10:04:42.227 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000001.91.5, , , , 00000000001.110.5, , , 00000000001.68.5, , 00000000001.40.5] 2021-02-11T10:04:42.227 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:04:42.224+0000 7fde050eb080 0 bucket checkpoint complete 2021-02-11T10:04:42.263 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-69', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:42.263 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-69', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:42.263 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-69 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:04:42.308 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:42.308 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:42.530 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:04:42.524+0000 7fac00930080 1 bucket sync caught up with source: 2021-02-11T10:04:42.530 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000001.91.5, , , , 00000000001.110.5, , , 00000000001.68.5, , 00000000001.40.5] 2021-02-11T10:04:42.530 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000001.91.5, , , , 00000000001.110.5, , , 00000000001.68.5, , 00000000001.40.5] 2021-02-11T10:04:42.530 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:04:42.524+0000 7fac00930080 0 bucket checkpoint complete 2021-02-11T10:04:42.565 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-69', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:42.565 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-69', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:42.566 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-69 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:04:42.613 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:42.613 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:42.842 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:04:42.836+0000 7f2dd3cf4080 1 bucket sync caught up with source: 2021-02-11T10:04:42.842 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000001.86.5, , , , 00000000001.90.5, , , 00000000001.67.5, , 00000000001.41.5] 2021-02-11T10:04:42.842 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000001.86.5, , , , 00000000001.90.5, , , 00000000001.67.5, , 00000000001.41.5] 2021-02-11T10:04:42.842 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:04:42.836+0000 7f2dd3cf4080 0 bucket checkpoint complete 2021-02-11T10:04:42.872 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-69', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:42.872 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-69', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:42.872 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-69 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:04:42.919 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:42.919 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:43.157 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:04:43.148+0000 7fc1bf19c080 1 bucket sync caught up with source: 2021-02-11T10:04:43.158 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, 00000000001.86.5, , , , 00000000001.90.5, , , 00000000001.67.5, , 00000000001.41.5] 2021-02-11T10:04:43.158 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, 00000000001.86.5, , , , 00000000001.90.5, , , 00000000001.67.5, , 00000000001.41.5] 2021-02-11T10:04:43.158 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:04:43.148+0000 7fc1bf19c080 0 bucket checkpoint complete 2021-02-11T10:04:43.194 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-69 zones={test-zone2, test-zone1} 2021-02-11T10:04:43.203 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:04:43.251 DEBUG:tasks.rgw_multi.zone_rados:o=a 2021-02-11T10:04:43.251 DEBUG:tasks.rgw_multi.zone_rados:o=b 2021-02-11T10:04:43.251 DEBUG:tasks.rgw_multi.zone_rados:o=c 2021-02-11T10:04:43.251 DEBUG:tasks.rgw_multi.zone_rados:o=d 2021-02-11T10:04:43.251 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:04:43.259 DEBUG:tasks.rgw_multi.zone_rados:o=a 2021-02-11T10:04:43.259 DEBUG:tasks.rgw_multi.zone_rados:o=b 2021-02-11T10:04:43.259 DEBUG:tasks.rgw_multi.zone_rados:o=c 2021-02-11T10:04:43.259 DEBUG:tasks.rgw_multi.zone_rados:o=d 2021-02-11T10:04:43.315 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-02-11T10:04:43.371 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-02-11T10:04:43.480 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-02-11T10:04:43.585 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-02-11T10:04:43.639 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-02-11T10:04:43.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-02-11T10:04:43.857 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-02-11T10:04:43.911 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-02-11T10:04:44.024 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-02-11T10:04:44.130 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-02-11T10:04:44.184 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-02-11T10:04:44.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-02-11T10:04:44.403 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-69 zones={test-zone2, test-zone1} 2021-02-11T10:04:44.403 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-69 zones={test-zone3, test-zone1} 2021-02-11T10:04:44.411 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:04:44.416 DEBUG:tasks.rgw_multi.zone_rados:o=a 2021-02-11T10:04:44.416 DEBUG:tasks.rgw_multi.zone_rados:o=b 2021-02-11T10:04:44.416 DEBUG:tasks.rgw_multi.zone_rados:o=c 2021-02-11T10:04:44.416 DEBUG:tasks.rgw_multi.zone_rados:o=d 2021-02-11T10:04:44.416 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:04:44.421 DEBUG:tasks.rgw_multi.zone_rados:o=a 2021-02-11T10:04:44.421 DEBUG:tasks.rgw_multi.zone_rados:o=b 2021-02-11T10:04:44.421 DEBUG:tasks.rgw_multi.zone_rados:o=c 2021-02-11T10:04:44.421 DEBUG:tasks.rgw_multi.zone_rados:o=d 2021-02-11T10:04:44.471 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-02-11T10:04:44.524 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-02-11T10:04:44.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-02-11T10:04:44.778 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-02-11T10:04:44.832 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-02-11T10:04:44.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-02-11T10:04:45.048 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-02-11T10:04:45.103 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-02-11T10:04:45.213 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-02-11T10:04:45.321 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-02-11T10:04:45.374 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-02-11T10:04:45.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-02-11T10:04:45.585 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-69 zones={test-zone3, test-zone1} 2021-02-11T10:04:45.586 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-69 zones={test-zone3, test-zone2} 2021-02-11T10:04:45.596 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:04:45.643 DEBUG:tasks.rgw_multi.zone_rados:o=a 2021-02-11T10:04:45.643 DEBUG:tasks.rgw_multi.zone_rados:o=b 2021-02-11T10:04:45.643 DEBUG:tasks.rgw_multi.zone_rados:o=c 2021-02-11T10:04:45.643 DEBUG:tasks.rgw_multi.zone_rados:o=d 2021-02-11T10:04:45.643 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:04:45.651 DEBUG:tasks.rgw_multi.zone_rados:o=a 2021-02-11T10:04:45.651 DEBUG:tasks.rgw_multi.zone_rados:o=b 2021-02-11T10:04:45.651 DEBUG:tasks.rgw_multi.zone_rados:o=c 2021-02-11T10:04:45.651 DEBUG:tasks.rgw_multi.zone_rados:o=d 2021-02-11T10:04:45.710 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-02-11T10:04:45.768 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-02-11T10:04:45.879 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-02-11T10:04:45.986 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-02-11T10:04:46.039 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-02-11T10:04:46.151 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-02-11T10:04:46.254 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-02-11T10:04:46.307 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-02-11T10:04:46.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-02-11T10:04:46.526 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-02-11T10:04:46.579 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-02-11T10:04:46.696 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-02-11T10:04:46.802 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-69 zones={test-zone3, test-zone2} 2021-02-11T10:04:46.802 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:04:46.802 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:04:46.802 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 0 --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T10:04:46.841 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:46.841 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:47.099 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=tkscyf-70 2021-02-11T10:04:47.338 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:47.339 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:47.339 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:04:47.378 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:47.378 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:47.546 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 2021-02-11T10:04:47.574 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613037887.134046_636.1","last_update":"2021-02-11T10:04:47.134046Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:04:47.574 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T10:04:47.575 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:47.575 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:47.575 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:04:47.616 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:47.616 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:47.838 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037887.134046_636.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:04:47.134046Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:04:47.838 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T10:04:47.838 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T10:04:47.838 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:47.838 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:47.839 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:04:47.875 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:47.875 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:48.098 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037887.134046_636.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:04:47.134046Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:04:48.099 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T10:04:48.099 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:04:48.099 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:48.099 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:48.099 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:04:48.138 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:48.138 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:48.376 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037887.134046_636.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:04:47.134046Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:04:48.376 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:04:48.377 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-70', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:48.377 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-70', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:48.377 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-70 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:04:48.463 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:48.463 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:48.700 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:04:48.691+0000 7f1381f82080 1 bucket sync caught up with source: 2021-02-11T10:04:48.700 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, 00000000001.48.5, , , , 00000000001.99.5, , , 00000000001.73.5, , 00000000001.67.5] 2021-02-11T10:04:48.700 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, 00000000001.48.5, , , , 00000000001.99.5, , , 00000000001.73.5, , 00000000001.67.5] 2021-02-11T10:04:48.700 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:04:48.691+0000 7f1381f82080 0 bucket checkpoint complete 2021-02-11T10:04:48.737 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-70', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:48.737 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-70', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:48.738 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-70 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:04:48.785 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:48.786 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:48.994 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:04:48.988+0000 7f65b78fa080 1 bucket sync caught up with source: 2021-02-11T10:04:48.994 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000001.48.5, , , , 00000000001.99.5, , , 00000000001.73.5, , 00000000001.67.5] 2021-02-11T10:04:48.994 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000001.48.5, , , , 00000000001.99.5, , , 00000000001.73.5, , 00000000001.67.5] 2021-02-11T10:04:48.994 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:04:48.988+0000 7f65b78fa080 0 bucket checkpoint complete 2021-02-11T10:04:49.022 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-70', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:49.022 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-70', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:49.022 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-70 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:04:49.071 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:49.071 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:49.293 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:04:49.287+0000 7fd87dddc080 1 bucket sync caught up with source: 2021-02-11T10:04:49.294 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, 00000000001.48.5, , , , 00000000001.99.5, , , 00000000001.73.5, , 00000000001.67.5] 2021-02-11T10:04:49.294 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, 00000000001.48.5, , , , 00000000001.99.5, , , 00000000001.73.5, , 00000000001.67.5] 2021-02-11T10:04:49.294 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:04:49.287+0000 7fd87dddc080 0 bucket checkpoint complete 2021-02-11T10:04:49.331 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-70', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:49.331 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-70', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:49.331 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-70 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:04:49.378 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:49.378 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:49.613 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:04:49.608+0000 7ffba2ea8080 1 bucket sync caught up with source: 2021-02-11T10:04:49.613 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000001.50.5, , , , 00000000001.95.5, , , 00000000001.57.5, , 00000000001.57.5] 2021-02-11T10:04:49.613 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000001.50.5, , , , 00000000001.95.5, , , 00000000001.57.5, , 00000000001.57.5] 2021-02-11T10:04:49.613 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:04:49.608+0000 7ffba2ea8080 0 bucket checkpoint complete 2021-02-11T10:04:49.646 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-70', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:49.647 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-70', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:49.647 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-70 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:04:49.689 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:49.689 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:49.917 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:04:49.912+0000 7fb558b9e080 1 bucket sync caught up with source: 2021-02-11T10:04:49.917 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000001.50.5, , , , 00000000001.95.5, , , 00000000001.57.5, , 00000000001.57.5] 2021-02-11T10:04:49.917 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000001.50.5, , , , 00000000001.95.5, , , 00000000001.57.5, , 00000000001.57.5] 2021-02-11T10:04:49.917 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:04:49.912+0000 7fb558b9e080 0 bucket checkpoint complete 2021-02-11T10:04:49.950 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-70', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:49.950 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-70', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:49.950 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-70 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:04:50.047 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:50.047 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:50.253 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:04:50.248+0000 7f905f1dd080 1 bucket sync caught up with source: 2021-02-11T10:04:50.253 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, 00000000001.51.5, , , , 00000000001.90.5, , , 00000000001.69.5, , 00000000001.58.5] 2021-02-11T10:04:50.253 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, 00000000001.51.5, , , , 00000000001.90.5, , , 00000000001.69.5, , 00000000001.58.5] 2021-02-11T10:04:50.253 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:04:50.248+0000 7f905f1dd080 0 bucket checkpoint complete 2021-02-11T10:04:50.285 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-70', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:50.285 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-70', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:04:50.285 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-70 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:04:50.335 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:50.336 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:50.555 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:04:50.547+0000 7fd2e27af080 1 bucket sync caught up with source: 2021-02-11T10:04:50.555 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, 00000000001.51.5, , , , 00000000001.90.5, , , 00000000001.69.5, , 00000000001.58.5] 2021-02-11T10:04:50.555 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, 00000000001.51.5, , , , 00000000001.90.5, , , 00000000001.69.5, , 00000000001.58.5] 2021-02-11T10:04:50.555 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:04:50.547+0000 7fd2e27af080 0 bucket checkpoint complete 2021-02-11T10:04:50.593 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-70 zones={test-zone2, test-zone1} 2021-02-11T10:04:50.604 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:04:50.610 DEBUG:tasks.rgw_multi.zone_rados:o=a 2021-02-11T10:04:50.611 DEBUG:tasks.rgw_multi.zone_rados:o=b 2021-02-11T10:04:50.611 DEBUG:tasks.rgw_multi.zone_rados:o=c 2021-02-11T10:04:50.611 DEBUG:tasks.rgw_multi.zone_rados:o=d 2021-02-11T10:04:50.611 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:04:50.617 DEBUG:tasks.rgw_multi.zone_rados:o=a 2021-02-11T10:04:50.617 DEBUG:tasks.rgw_multi.zone_rados:o=b 2021-02-11T10:04:50.617 DEBUG:tasks.rgw_multi.zone_rados:o=c 2021-02-11T10:04:50.617 DEBUG:tasks.rgw_multi.zone_rados:o=d 2021-02-11T10:04:50.628 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-02-11T10:04:50.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-02-11T10:04:50.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-02-11T10:04:50.897 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-02-11T10:04:50.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-02-11T10:04:51.068 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-02-11T10:04:51.174 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-02-11T10:04:51.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-02-11T10:04:51.334 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-02-11T10:04:51.438 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-02-11T10:04:51.492 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-02-11T10:04:51.604 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-02-11T10:04:51.709 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-70 zones={test-zone2, test-zone1} 2021-02-11T10:04:51.709 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-70 zones={test-zone3, test-zone1} 2021-02-11T10:04:51.718 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:04:51.723 DEBUG:tasks.rgw_multi.zone_rados:o=a 2021-02-11T10:04:51.723 DEBUG:tasks.rgw_multi.zone_rados:o=b 2021-02-11T10:04:51.723 DEBUG:tasks.rgw_multi.zone_rados:o=c 2021-02-11T10:04:51.723 DEBUG:tasks.rgw_multi.zone_rados:o=d 2021-02-11T10:04:51.723 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:04:51.727 DEBUG:tasks.rgw_multi.zone_rados:o=a 2021-02-11T10:04:51.727 DEBUG:tasks.rgw_multi.zone_rados:o=b 2021-02-11T10:04:51.728 DEBUG:tasks.rgw_multi.zone_rados:o=c 2021-02-11T10:04:51.728 DEBUG:tasks.rgw_multi.zone_rados:o=d 2021-02-11T10:04:51.736 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-02-11T10:04:51.790 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-02-11T10:04:51.899 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-02-11T10:04:52.002 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-02-11T10:04:52.055 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-02-11T10:04:52.163 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-02-11T10:04:52.264 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-02-11T10:04:52.321 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-02-11T10:04:52.426 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-02-11T10:04:52.529 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-02-11T10:04:52.583 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-02-11T10:04:52.696 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-02-11T10:04:52.801 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-70 zones={test-zone3, test-zone1} 2021-02-11T10:04:52.801 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-70 zones={test-zone3, test-zone2} 2021-02-11T10:04:52.812 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:04:52.816 DEBUG:tasks.rgw_multi.zone_rados:o=a 2021-02-11T10:04:52.816 DEBUG:tasks.rgw_multi.zone_rados:o=b 2021-02-11T10:04:52.817 DEBUG:tasks.rgw_multi.zone_rados:o=c 2021-02-11T10:04:52.817 DEBUG:tasks.rgw_multi.zone_rados:o=d 2021-02-11T10:04:52.817 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:04:52.823 DEBUG:tasks.rgw_multi.zone_rados:o=a 2021-02-11T10:04:52.823 DEBUG:tasks.rgw_multi.zone_rados:o=b 2021-02-11T10:04:52.823 DEBUG:tasks.rgw_multi.zone_rados:o=c 2021-02-11T10:04:52.823 DEBUG:tasks.rgw_multi.zone_rados:o=d 2021-02-11T10:04:52.875 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-02-11T10:04:52.928 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-02-11T10:04:53.036 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-02-11T10:04:53.142 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-02-11T10:04:53.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-02-11T10:04:53.306 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-02-11T10:04:53.404 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-02-11T10:04:53.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-02-11T10:04:53.565 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-02-11T10:04:53.674 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-02-11T10:04:53.727 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-02-11T10:04:53.839 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-02-11T10:04:53.982 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=tkscyf-70 zones={test-zone3, test-zone2} 2021-02-11T10:04:53.983 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '1', '--rgw-sync-log-trim-min-cold-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:04:53.983 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '1', '--rgw-sync-log-trim-min-cold-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:04:53.983 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 1 --rgw-sync-log-trim-min-cold-buckets 0 --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T10:04:54.021 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:54.021 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:54.293 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'tkscyf-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:54.294 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'tkscyf-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:54.294 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket tkscyf-70 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:04:54.374 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:54.374 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:54.605 INFO:teuthology.orchestra.run.smithi094.stdout:[] 2021-02-11T10:04:54.606 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'tkscyf-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:54.606 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'tkscyf-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:54.606 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket tkscyf-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:04:54.678 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:54.678 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:54.924 INFO:teuthology.orchestra.run.smithi094.stdout:[{"op_id":"1#00000000001.95.5","op_tag":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5482.4309","op":"write","object":"a","instance":"","state":"complete","index_ver":1,"timestamp":"2021-02-11T10:04:39.703761482Z","ver":{"pool":9,"epoch":16},"bilog_flags":0,"versioned":false,"owner":"","owner_display_name":"","zones_trace":[{"entry":"dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-69:dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.20"}]},{"op_id":"5#00000000001.74.5","op_tag":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5482.4324","op":"write","object":"d","instance":"","state":"complete","index_ver":1,"timestamp":"2021-02-11T10:04:39.853736006Z","ver":{"pool":9,"epoch":3},"bilog_flags":0,"versioned":false,"owner":"","owner_display_name":"","zones_trace":[{"entry":"dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-69:dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.20"}]},{"op_id":"8#00000000001.65.5","op_tag":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5482.4322","op":"write","object":"c","instance":"","state":"complete","index_ver":1,"timestamp":"2021-02-11T10:04:39.803163314Z","ver":{"pool":9,"epoch":3},"bilog_flags":0,"versioned":false,"owner":"","owner_display_name":"","zones_trace":[{"entry":"dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-69:dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.20"}]},{"op_id":"10#00000000001.46.5","op_tag":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5482.4320","op":"write","object":"b","instance":"","state":"complete","index_ver":1,"timestamp":"2021-02-11T10:04:39.751600418Z","ver":{"pool":9,"epoch":3},"bilog_flags":0,"versioned":false,"owner":"","owner_display_name":"","zones_trace":[{"entry":"dd308248-8ee8-49be-b928-07c3de7d7aa2:tkscyf-69:dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.20"}]}] 2021-02-11T10:04:54.924 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '999', '--rgw-sync-log-trim-min-cold-buckets', '999', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:04:54.924 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '999', '--rgw-sync-log-trim-min-cold-buckets', '999', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:04:54.925 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 999 --rgw-sync-log-trim-min-cold-buckets 999 --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T10:04:55.005 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:55.005 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:56.476 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'tkscyf-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:56.476 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'tkscyf-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:04:56.476 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket tkscyf-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:04:56.556 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:04:56.556 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:04:56.805 INFO:teuthology.orchestra.run.smithi094.stdout:[] 2021-02-11T10:04:56.806 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_index_log_trim ... ok 2021-02-11T10:04:56.846 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_creation_time ... ok 2021-02-11T10:04:56.846 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=tkscyf-71 2021-02-11T10:04:56.892 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=tkscyf-72 2021-02-11T10:04:56.979 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=tkscyf-73 2021-02-11T10:04:57.187 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:05:07.236 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:05:07.236 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:05:07.276 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:05:07.276 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:05:07.460 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 2021-02-11T10:05:07.497 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613037897.018955_662.1","last_update":"2021-02-11T10:04:57.018955Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:05:07.498 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T10:05:07.498 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:05:07.498 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:05:07.498 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:05:07.538 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:05:07.538 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:05:07.789 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037897.018955_662.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:04:57.018955Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:05:07.790 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T10:05:07.790 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T10:05:07.790 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:05:07.790 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:05:07.790 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:05:07.832 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:05:07.832 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:05:08.069 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037897.018955_662.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:04:57.018955Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:05:08.069 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T10:05:08.069 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:05:08.070 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:05:08.070 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:05:08.070 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:05:08.110 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:05:08.110 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:05:08.354 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037897.018955_662.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:04:57.018955Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:05:08.354 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:05:08.354 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:05:08.354 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:05:08.354 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:05:08.387 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:05:08.387 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:05:08.638 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613037887.303587_630.1","last_update":"2021-02-11T10:04:47.303587Z"},{"marker":"1_1613037887.201196_756.1","last_update":"2021-02-11T10:04:47.201196Z"},{"marker":"1_1613037897.529064_348.1","last_update":"2021-02-11T10:04:57.529064Z"},{"marker":"1_1613037887.252013_419.1","last_update":"2021-02-11T10:04:47.252013Z"}] 2021-02-11T10:05:08.638 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-02-11T10:05:08.638 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:05:08.638 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:05:08.639 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:05:08.681 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:05:08.682 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:05:08.938 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4703921989677143155},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037887.303587_630.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.303587Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037887.201196_756.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.201196Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613037897.529064_348.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:57.529064Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037887.252013_419.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.252013Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:05:08.939 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-02-11T10:05:08.939 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:05:08.939 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:05:08.939 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:05:08.980 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:05:08.980 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:05:09.237 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613037887.303587_630.1","last_update":"2021-02-11T10:04:47.303587Z"},{"marker":"1_1613037887.201196_756.1","last_update":"2021-02-11T10:04:47.201196Z"},{"marker":"1_1613037897.529064_348.1","last_update":"2021-02-11T10:04:57.529064Z"},{"marker":"1_1613037887.252013_419.1","last_update":"2021-02-11T10:04:47.252013Z"}] 2021-02-11T10:05:09.237 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-02-11T10:05:09.237 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:05:09.237 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:05:09.238 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:05:09.328 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:05:09.328 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:05:09.558 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":13312279424886386302},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037887.303587_630.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.303587Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037887.201196_756.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.201196Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613037897.529064_348.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:57.529064Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037887.252013_419.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.252013Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:05:09.558 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-02-11T10:05:09.558 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:05:09.558 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:05:09.559 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:05:09.594 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:05:09.595 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:05:09.850 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613037887.303587_630.1","last_update":"2021-02-11T10:04:47.303587Z"},{"marker":"1_1613037887.201196_756.1","last_update":"2021-02-11T10:04:47.201196Z"},{"marker":"1_1613037897.529064_348.1","last_update":"2021-02-11T10:04:57.529064Z"},{"marker":"1_1613037887.252013_419.1","last_update":"2021-02-11T10:04:47.252013Z"}] 2021-02-11T10:05:09.850 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-02-11T10:05:09.851 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:05:09.851 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:05:09.851 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:05:09.894 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:05:09.894 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:05:10.145 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2896161496309628191},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037887.303587_630.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.303587Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037887.201196_756.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.201196Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613037897.529064_348.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:57.529064Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037887.252013_419.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.252013Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:05:10.146 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-02-11T10:05:10.146 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:05:10.146 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:05:10.146 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:05:10.230 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:05:10.230 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:05:10.484 INFO:teuthology.orchestra.run.smithi098.stdout:[{"marker":"1_1613037887.475490_256.1","last_update":"2021-02-11T10:04:47.475490Z"},{"marker":"1_1613037887.423921_672.1","last_update":"2021-02-11T10:04:47.423921Z"},{"marker":"1_1613037897.493982_336.1","last_update":"2021-02-11T10:04:57.493982Z"},{"marker":"1_1613037887.524245_409.1","last_update":"2021-02-11T10:04:47.524245Z"}] 2021-02-11T10:05:10.485 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-02-11T10:05:10.485 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:05:10.485 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:05:10.485 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:05:10.527 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:05:10.527 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:05:10.778 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":15226599365335882107},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037887.475490_256.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.475490Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037887.423921_672.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.423921Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613037897.493982_336.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:57.493982Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037887.524245_409.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.524245Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:05:10.778 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-02-11T10:05:10.778 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:05:10.778 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:05:10.779 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:05:10.818 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:05:10.818 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:05:11.069 INFO:teuthology.orchestra.run.smithi098.stdout:[{"marker":"1_1613037887.475490_256.1","last_update":"2021-02-11T10:04:47.475490Z"},{"marker":"1_1613037887.423921_672.1","last_update":"2021-02-11T10:04:47.423921Z"},{"marker":"1_1613037897.493982_336.1","last_update":"2021-02-11T10:04:57.493982Z"},{"marker":"1_1613037887.524245_409.1","last_update":"2021-02-11T10:04:47.524245Z"}] 2021-02-11T10:05:11.069 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-02-11T10:05:11.069 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:05:11.069 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:05:11.070 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:05:11.111 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:05:11.111 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:05:11.359 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":10999913493642558030},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037887.475490_256.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.475490Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037887.423921_672.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.423921Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613037897.493982_336.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:57.493982Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037887.524245_409.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.524245Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:05:11.360 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-02-11T10:05:11.360 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:05:11.360 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:05:11.360 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:05:11.441 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:05:11.441 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:05:11.686 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613037887.476185_349.1","last_update":"2021-02-11T10:04:47.476185Z"},{"marker":"1_1613037887.464090_596.1","last_update":"2021-02-11T10:04:47.464090Z"},{"marker":"1_1613037897.275075_252.1","last_update":"2021-02-11T10:04:57.275075Z"},{"marker":"1_1613037887.474519_308.1","last_update":"2021-02-11T10:04:47.474519Z"}] 2021-02-11T10:05:11.687 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-02-11T10:05:11.687 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:05:11.687 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:05:11.687 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:05:11.773 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:05:11.773 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:05:12.026 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2753764877348348355},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037887.476185_349.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.476185Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037887.464090_596.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.464090Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613037897.275075_252.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:57.275075Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037887.474519_308.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.474519Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:05:12.026 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-02-11T10:05:12.026 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:05:12.027 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:05:12.027 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:05:12.108 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:05:12.108 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:05:12.355 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613037887.476185_349.1","last_update":"2021-02-11T10:04:47.476185Z"},{"marker":"1_1613037887.464090_596.1","last_update":"2021-02-11T10:04:47.464090Z"},{"marker":"1_1613037897.275075_252.1","last_update":"2021-02-11T10:04:57.275075Z"},{"marker":"1_1613037887.474519_308.1","last_update":"2021-02-11T10:04:47.474519Z"}] 2021-02-11T10:05:12.355 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-02-11T10:05:12.355 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:05:12.355 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:05:12.355 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:05:12.404 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:05:12.404 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:05:12.654 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":7143279782539285748},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037887.476185_349.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.476185Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037887.464090_596.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.464090Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613037897.275075_252.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:57.275075Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037887.474519_308.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.474519Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:05:12.655 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-02-11T10:05:12.655 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'type', '--log-type', 'fifo', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:05:12.655 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'type', '--log-type', 'fifo', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:05:12.655 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog type --log-type fifo --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T10:05:12.693 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:05:12.693 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:05:13.003 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'type', '--log-type', 'fifo', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:05:13.003 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'type', '--log-type', 'fifo', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:05:13.003 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog type --log-type fifo --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T10:05:13.040 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:05:13.040 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:05:13.343 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'type', '--log-type', 'fifo', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:05:13.343 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'type', '--log-type', 'fifo', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:05:13.343 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog type --log-type fifo --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T10:05:13.376 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:05:13.376 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:05:13.639 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:05:13.639 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:05:13.639 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:05:13.677 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:05:13.677 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:05:13.880 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 2021-02-11T10:05:13.921 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613037897.018955_662.1","last_update":"2021-02-11T10:04:57.018955Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:05:13.922 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T10:05:13.922 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:05:13.922 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:05:13.922 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:05:13.964 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:05:13.964 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:05:14.220 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037897.018955_662.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:04:57.018955Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:05:14.220 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T10:05:14.220 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T10:05:14.220 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:05:14.221 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:05:14.221 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:05:14.261 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:05:14.262 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:05:14.503 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037897.018955_662.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:04:57.018955Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:05:14.503 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T10:05:14.503 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:05:14.503 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:05:14.503 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:05:14.503 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:05:14.544 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:05:14.545 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:05:14.793 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037897.018955_662.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:04:57.018955Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:05:14.793 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:05:14.793 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:05:14.793 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:05:14.794 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:05:14.832 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:05:14.832 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:05:15.094 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"G00000000000000000001@00000000000000000000:00000000000000000000","last_update":"0.000000"},{"marker":"G00000000000000000001@00000000000000000000:00000000000000000000","last_update":"0.000000"},{"marker":"G00000000000000000001@00000000000000000000:00000000000000000000","last_update":"0.000000"},{"marker":"G00000000000000000001@00000000000000000000:00000000000000000000","last_update":"0.000000"}] 2021-02-11T10:05:15.094 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-02-11T10:05:15.094 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:05:15.094 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:05:15.094 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:05:15.136 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:05:15.137 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:05:15.386 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4703921989677143155},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037887.303587_630.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.303587Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037887.201196_756.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.201196Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613037897.529064_348.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:57.529064Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037887.252013_419.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.252013Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:05:15.386 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.303587_630.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.201196_756.1, shard=2 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037897.529064_348.1, shard=3 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.252013_419.1 2021-02-11T10:05:20.387 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:05:20.387 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:05:20.388 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:05:20.429 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:05:20.429 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:05:20.704 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4703921989677143155},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037887.303587_630.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.303587Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037887.201196_756.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.201196Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613037897.529064_348.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:57.529064Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037887.252013_419.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.252013Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:05:20.705 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.303587_630.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.201196_756.1, shard=2 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037897.529064_348.1, shard=3 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.252013_419.1 2021-02-11T10:05:25.706 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:05:25.706 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:05:25.706 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:05:25.749 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:05:25.749 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:05:26.019 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4703921989677143155},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037887.303587_630.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.303587Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037887.201196_756.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.201196Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1613037897.529064_348.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:57.529064Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037887.252013_419.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.252013Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:05:26.020 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.303587_630.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.201196_756.1, shard=2 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037897.529064_348.1, shard=3 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.252013_419.1 2021-02-11T10:05:31.020 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:05:40.575 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:05:40.576 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:05:40.619 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:05:40.619 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:05:40.888 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4703921989677143155},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037887.303587_630.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.303587Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037887.201196_756.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.201196Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:28.881878Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037887.252013_419.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.252013Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:05:40.889 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.303587_630.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.201196_756.1, shard=3 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.252013_419.1 2021-02-11T10:05:45.889 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:05:45.890 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:05:45.890 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:05:45.934 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:05:45.934 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:05:46.213 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4703921989677143155},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037887.303587_630.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.303587Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037887.201196_756.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.201196Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:28.881878Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037887.252013_419.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.252013Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:05:46.213 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.303587_630.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.201196_756.1, shard=3 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.252013_419.1 2021-02-11T10:05:51.214 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:05:51.215 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:05:51.215 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:05:51.257 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:05:51.257 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:05:51.536 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4703921989677143155},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037887.303587_630.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.303587Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037887.201196_756.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.201196Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:28.881878Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037887.252013_419.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.252013Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:05:51.537 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.303587_630.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.201196_756.1, shard=3 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.252013_419.1 2021-02-11T10:05:56.538 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:05:56.538 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:05:56.539 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:05:56.576 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:05:56.577 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:05:56.838 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4703921989677143155},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037887.303587_630.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.303587Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037887.201196_756.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.201196Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:28.881878Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037887.252013_419.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.252013Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:05:56.838 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.303587_630.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.201196_756.1, shard=3 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.252013_419.1 2021-02-11T10:06:01.839 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:06:01.840 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:06:01.840 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:06:01.883 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:06:01.883 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:06:02.153 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4703921989677143155},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037887.303587_630.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.303587Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037887.201196_756.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.201196Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:28.881878Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037887.252013_419.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.252013Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:06:02.153 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.303587_630.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.201196_756.1, shard=3 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.252013_419.1 2021-02-11T10:06:07.154 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:06:32.736 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:06:32.736 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:06:32.780 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:06:32.780 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:06:33.030 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4703921989677143155},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037887.303587_630.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.303587Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037887.201196_756.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.201196Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:28.881878Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037887.252013_419.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.252013Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:06:33.031 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.303587_630.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.201196_756.1, shard=3 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.252013_419.1 2021-02-11T10:06:38.032 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:06:38.032 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:06:38.032 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:06:38.076 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:06:38.076 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:06:38.340 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4703921989677143155},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037887.303587_630.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.303587Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037887.201196_756.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.201196Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:28.881878Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037887.252013_419.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.252013Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:06:38.340 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.303587_630.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.201196_756.1, shard=3 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.252013_419.1 2021-02-11T10:06:43.341 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:06:54.209 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:06:54.210 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:06:54.252 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:06:54.252 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:06:54.526 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4703921989677143155},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037887.303587_630.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.303587Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037887.201196_756.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.201196Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:28.881878Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037887.252013_419.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.252013Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:06:54.527 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.303587_630.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.201196_756.1, shard=3 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.252013_419.1 2021-02-11T10:06:59.528 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:06:59.528 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:06:59.529 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:06:59.570 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:06:59.570 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:06:59.834 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4703921989677143155},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037887.303587_630.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.303587Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037887.201196_756.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.201196Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:28.881878Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037887.252013_419.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.252013Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:06:59.835 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.303587_630.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.201196_756.1, shard=3 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.252013_419.1 2021-02-11T10:07:04.835 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:07:04.836 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:07:04.836 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:07:04.877 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:07:04.877 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:07:05.185 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4703921989677143155},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037887.303587_630.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.303587Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037887.201196_756.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.201196Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:28.881878Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037887.252013_419.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.252013Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:07:05.185 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.303587_630.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.201196_756.1, shard=3 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.252013_419.1 2021-02-11T10:07:10.186 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:07:10.186 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:07:10.186 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:07:10.230 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:07:10.231 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:07:10.497 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4703921989677143155},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037887.303587_630.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.303587Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037887.201196_756.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.201196Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:28.881878Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037887.252013_419.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.252013Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:07:10.497 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.303587_630.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.201196_756.1, shard=3 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.252013_419.1 2021-02-11T10:07:15.498 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:07:15.499 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:07:15.499 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:07:15.540 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:07:15.540 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:07:15.822 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4703921989677143155},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037887.303587_630.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.303587Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037887.201196_756.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.201196Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:28.881878Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037887.252013_419.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.252013Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:07:15.823 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.303587_630.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.201196_756.1, shard=3 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.252013_419.1 2021-02-11T10:07:20.824 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:07:20.824 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:07:20.825 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:07:20.868 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:07:20.868 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:07:21.133 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4703921989677143155},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037887.303587_630.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.303587Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037887.201196_756.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.201196Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:28.881878Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037887.252013_419.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.252013Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:07:21.134 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.303587_630.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.201196_756.1, shard=3 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.252013_419.1 2021-02-11T10:07:26.134 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:07:26.135 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:07:26.135 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:07:26.178 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:07:26.179 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:07:26.438 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4703921989677143155},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037887.303587_630.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.303587Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037887.201196_756.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.201196Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:28.881878Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037887.252013_419.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.252013Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:07:26.438 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.303587_630.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.201196_756.1, shard=3 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.252013_419.1 2021-02-11T10:07:31.440 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:07:31.440 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:07:31.440 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:07:31.491 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:07:31.491 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:07:31.763 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4703921989677143155},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037887.303587_630.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.303587Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037887.201196_756.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.201196Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:28.881878Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037887.252013_419.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.252013Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:07:31.764 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.303587_630.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.201196_756.1, shard=3 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.252013_419.1 2021-02-11T10:07:36.764 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:07:36.765 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:07:36.765 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:07:36.816 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:07:36.816 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:07:37.081 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4703921989677143155},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037887.303587_630.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.303587Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037887.201196_756.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.201196Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:28.881878Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037887.252013_419.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.252013Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:07:37.081 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.303587_630.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.201196_756.1, shard=3 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.252013_419.1 2021-02-11T10:07:42.082 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:07:42.083 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:07:42.083 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:07:42.126 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:07:42.127 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:07:42.403 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4703921989677143155},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037887.303587_630.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.303587Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037887.201196_756.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.201196Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:28.881878Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037887.252013_419.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.252013Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:07:42.404 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.303587_630.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.201196_756.1, shard=3 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.252013_419.1 2021-02-11T10:07:47.405 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:07:47.405 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:07:47.406 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:07:47.448 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:07:47.448 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:07:47.693 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4703921989677143155},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037887.303587_630.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.303587Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037887.201196_756.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.201196Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:28.881878Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037887.252013_419.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.252013Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:07:47.693 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.303587_630.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.201196_756.1, shard=3 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.252013_419.1 2021-02-11T10:07:52.695 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:00.668 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:00.668 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:08:00.712 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:08:00.712 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:08:01.000 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4703921989677143155},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037887.303587_630.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.303587Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037887.201196_756.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.201196Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:28.881878Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037887.252013_419.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.252013Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:08:01.000 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.303587_630.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.201196_756.1, shard=3 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.252013_419.1 2021-02-11T10:08:06.001 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:06.001 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:06.001 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:08:06.044 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:08:06.044 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:08:06.309 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4703921989677143155},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037887.303587_630.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.303587Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037887.201196_756.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.201196Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:28.881878Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037887.252013_419.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.252013Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:08:06.309 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.303587_630.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.201196_756.1, shard=3 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.252013_419.1 2021-02-11T10:08:11.311 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:11.311 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:11.311 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:08:11.353 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:08:11.353 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:08:11.613 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4703921989677143155},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037887.303587_630.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.303587Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037887.201196_756.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.201196Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:28.881878Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037887.252013_419.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.252013Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:08:11.614 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.303587_630.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.201196_756.1, shard=3 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.252013_419.1 2021-02-11T10:08:16.614 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:16.615 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:16.615 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:08:16.657 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:08:16.657 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:08:16.912 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4703921989677143155},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037887.303587_630.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.303587Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037887.201196_756.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.201196Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:28.881878Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037887.252013_419.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.252013Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:08:16.912 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.303587_630.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.201196_756.1, shard=3 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.252013_419.1 2021-02-11T10:08:21.914 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:21.914 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:21.914 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:08:21.956 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:08:21.957 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:08:22.226 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4703921989677143155},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037887.303587_630.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.303587Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037887.201196_756.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.201196Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:28.881878Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037887.252013_419.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.252013Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:08:22.226 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.303587_630.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.201196_756.1, shard=3 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.252013_419.1 2021-02-11T10:08:27.227 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:27.228 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:27.228 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:08:27.271 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:08:27.272 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:08:27.537 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4703921989677143155},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037887.303587_630.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.303587Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037887.201196_756.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.201196Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:28.881878Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037887.252013_419.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.252013Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:08:27.538 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.303587_630.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.201196_756.1, shard=3 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.252013_419.1 2021-02-11T10:08:32.539 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:32.539 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:32.539 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:08:32.581 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:08:32.581 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:08:32.852 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4703921989677143155},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037887.303587_630.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.303587Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037887.201196_756.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.201196Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:28.881878Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037887.252013_419.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.252013Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:08:32.853 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.303587_630.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.201196_756.1, shard=3 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.252013_419.1 2021-02-11T10:08:37.853 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:37.854 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:37.854 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:08:37.896 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:08:37.896 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:08:38.162 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4703921989677143155},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037887.303587_630.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.303587Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037887.201196_756.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.201196Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:28.881878Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037887.252013_419.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.252013Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:08:38.163 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.303587_630.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.201196_756.1, shard=3 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.252013_419.1 2021-02-11T10:08:43.163 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:43.164 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:43.164 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:08:43.207 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:08:43.207 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:08:43.461 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4703921989677143155},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037887.303587_630.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.303587Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037887.201196_756.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.201196Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:28.881878Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037887.252013_419.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.252013Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:08:43.462 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.303587_630.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.201196_756.1, shard=3 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.252013_419.1 2021-02-11T10:08:48.463 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:48.463 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:48.464 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:08:48.506 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:08:48.506 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:08:48.772 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4703921989677143155},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037887.303587_630.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.303587Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037887.201196_756.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.201196Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:28.881878Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037887.252013_419.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.252013Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:08:48.773 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.303587_630.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.201196_756.1, shard=3 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.252013_419.1 2021-02-11T10:08:53.773 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:53.774 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:08:53.774 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:08:53.820 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:08:53.820 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:08:54.090 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4703921989677143155},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037887.303587_630.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.303587Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037887.201196_756.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.201196Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:28.881878Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037887.252013_419.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.252013Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:08:54.091 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.303587_630.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.201196_756.1, shard=3 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.252013_419.1 2021-02-11T10:08:59.091 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:12.114 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:12.114 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:09:12.156 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:09:12.156 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:09:12.411 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4703921989677143155},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037887.303587_630.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.303587Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037887.201196_756.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.201196Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:28.881878Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037887.252013_419.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.252013Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:09:12.411 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.303587_630.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.201196_756.1, shard=3 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.252013_419.1 2021-02-11T10:09:17.412 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:17.412 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:17.413 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:09:17.450 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:09:17.450 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:09:17.728 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4703921989677143155},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037887.303587_630.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.303587Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037887.201196_756.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.201196Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:28.881878Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037887.252013_419.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.252013Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:09:17.728 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.303587_630.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.201196_756.1, shard=3 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.252013_419.1 2021-02-11T10:09:22.730 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:22.730 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:22.730 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:09:22.773 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:09:22.773 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:09:23.042 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4703921989677143155},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037887.303587_630.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.303587Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037887.201196_756.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.201196Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:28.881878Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037887.252013_419.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.252013Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:09:23.043 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.303587_630.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.201196_756.1, shard=3 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.252013_419.1 2021-02-11T10:09:28.044 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:28.045 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:28.045 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:09:28.087 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:09:28.087 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:09:28.371 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4703921989677143155},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037887.303587_630.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.303587Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037887.201196_756.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.201196Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:28.881878Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037887.252013_419.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.252013Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:09:28.372 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.303587_630.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.201196_756.1, shard=3 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.252013_419.1 2021-02-11T10:09:33.372 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:38.036 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:38.037 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:09:38.080 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:09:38.080 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:09:38.355 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4703921989677143155},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037887.303587_630.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.303587Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037887.201196_756.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.201196Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:28.881878Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037887.252013_419.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.252013Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:09:38.355 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.303587_630.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.201196_756.1, shard=3 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.252013_419.1 2021-02-11T10:09:43.355 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:43.356 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:43.356 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:09:43.399 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:09:43.399 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:09:43.648 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4703921989677143155},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037887.303587_630.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.303587Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037887.201196_756.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.201196Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:28.881878Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037887.252013_419.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.252013Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:09:43.648 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.303587_630.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.201196_756.1, shard=3 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.252013_419.1 2021-02-11T10:09:48.649 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:48.649 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:48.649 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:09:48.693 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:09:48.693 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:09:48.979 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4703921989677143155},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037887.303587_630.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.303587Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037887.201196_756.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.201196Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:28.881878Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037887.252013_419.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.252013Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:09:48.979 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.303587_630.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.201196_756.1, shard=3 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.252013_419.1 2021-02-11T10:09:53.981 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:53.981 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:53.981 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:09:54.024 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:09:54.024 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:09:54.303 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4703921989677143155},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037887.303587_630.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.303587Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037887.201196_756.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.201196Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:28.881878Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037887.252013_419.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.252013Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:09:54.303 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.303587_630.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.201196_756.1, shard=3 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.252013_419.1 2021-02-11T10:09:59.304 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:59.304 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:09:59.305 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:09:59.348 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:09:59.348 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:09:59.617 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4703921989677143155},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037887.303587_630.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.303587Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037887.201196_756.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.201196Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:28.881878Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037887.252013_419.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.252013Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:09:59.617 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.303587_630.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.201196_756.1, shard=3 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.252013_419.1 2021-02-11T10:10:04.618 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:10:04.619 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:10:04.619 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:10:04.660 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:10:04.661 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:10:04.928 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4703921989677143155},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037887.303587_630.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.303587Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037887.201196_756.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.201196Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:28.881878Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037887.252013_419.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.252013Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:10:04.928 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.303587_630.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.201196_756.1, shard=3 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.252013_419.1 2021-02-11T10:10:09.929 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:10:31.757 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:10:31.757 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:10:31.800 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:10:31.801 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:10:32.053 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4703921989677143155},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037887.303587_630.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.303587Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037887.201196_756.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.201196Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:28.881878Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037887.252013_419.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.252013Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:10:32.054 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.303587_630.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.201196_756.1, shard=3 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.252013_419.1 2021-02-11T10:10:37.054 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:10:37.054 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:10:37.054 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:10:37.097 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:10:37.097 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:10:37.363 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4703921989677143155},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037887.303587_630.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.303587Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037887.201196_756.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.201196Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:28.881878Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037887.252013_419.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.252013Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:10:37.363 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.303587_630.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.201196_756.1, shard=3 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.252013_419.1 2021-02-11T10:10:42.364 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:10:51.794 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:10:51.794 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:10:51.837 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:10:51.837 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:10:52.094 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4703921989677143155},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037887.303587_630.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.303587Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037887.201196_756.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.201196Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:28.881878Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037887.252013_419.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.252013Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:10:52.094 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.303587_630.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.201196_756.1, shard=3 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.252013_419.1 2021-02-11T10:10:57.095 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:10:57.095 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:10:57.095 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:10:57.137 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:10:57.138 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:10:57.401 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4703921989677143155},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037887.303587_630.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.303587Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037887.201196_756.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.201196Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:28.881878Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037887.252013_419.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.252013Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:10:57.402 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.303587_630.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.201196_756.1, shard=3 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.252013_419.1 2021-02-11T10:11:02.403 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:11:02.403 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:11:02.404 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:11:02.441 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:11:02.442 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:11:02.708 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4703921989677143155},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037887.303587_630.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.303587Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037887.201196_756.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.201196Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:28.881878Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037887.252013_419.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.252013Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:11:02.708 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.303587_630.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.201196_756.1, shard=3 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.252013_419.1 2021-02-11T10:11:07.709 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:11:07.709 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:11:07.709 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:11:07.753 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:11:07.753 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:11:08.032 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4703921989677143155},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037887.303587_630.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.303587Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037887.201196_756.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.201196Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:28.881878Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037887.252013_419.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.252013Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:11:08.032 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.303587_630.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.201196_756.1, shard=3 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.252013_419.1 2021-02-11T10:11:13.033 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:11:13.033 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:11:13.033 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:11:13.076 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:11:13.076 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:11:13.336 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4703921989677143155},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037887.303587_630.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.303587Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037887.201196_756.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.201196Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:28.881878Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037887.252013_419.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.252013Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:11:13.929 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.303587_630.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.201196_756.1, shard=3 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.252013_419.1 2021-02-11T10:11:18.930 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:11:18.930 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:11:18.930 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:11:18.974 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:11:18.974 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:11:19.248 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4703921989677143155},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037887.303587_630.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.303587Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037887.201196_756.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.201196Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:28.881878Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037887.252013_419.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.252013Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:11:19.248 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.303587_630.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.201196_756.1, shard=3 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.252013_419.1 2021-02-11T10:11:24.248 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:11:24.249 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:11:24.249 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:11:24.292 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:11:24.292 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:11:24.562 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4703921989677143155},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037887.303587_630.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.303587Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037887.201196_756.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.201196Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:28.881878Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037887.252013_419.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.252013Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:11:24.563 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.303587_630.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.201196_756.1, shard=3 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.252013_419.1 2021-02-11T10:11:29.564 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:11:29.564 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:11:29.564 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:11:29.607 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:11:29.607 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:11:29.867 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4703921989677143155},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037887.303587_630.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.303587Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037887.201196_756.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.201196Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:28.881878Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037887.252013_419.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.252013Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:11:29.868 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.303587_630.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.201196_756.1, shard=3 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.252013_419.1 2021-02-11T10:11:34.869 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:11:34.869 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:11:34.869 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:11:34.912 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:11:34.912 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:11:35.175 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4703921989677143155},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037887.303587_630.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.303587Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037887.201196_756.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.201196Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:28.881878Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037887.252013_419.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.252013Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:11:35.175 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.303587_630.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.201196_756.1, shard=3 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.252013_419.1 2021-02-11T10:11:40.176 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:11:40.176 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:11:40.177 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:11:40.221 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:11:40.221 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:11:40.498 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4703921989677143155},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037887.303587_630.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.303587Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037887.201196_756.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.201196Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:28.881878Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037887.252013_419.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.252013Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:11:40.499 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.303587_630.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.201196_756.1, shard=3 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.252013_419.1 2021-02-11T10:11:45.500 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:12:25.022 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:12:25.022 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:12:25.066 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:12:25.066 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:12:25.353 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4703921989677143155},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037887.303587_630.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.303587Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037887.201196_756.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.201196Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:28.881878Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037887.252013_419.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.252013Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:12:25.353 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.303587_630.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.201196_756.1, shard=3 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.252013_419.1 2021-02-11T10:12:30.354 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:12:30.354 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:12:30.354 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:12:30.398 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:12:30.398 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:12:30.672 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4703921989677143155},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037887.303587_630.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.303587Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037887.201196_756.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.201196Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:28.881878Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037887.252013_419.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.252013Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:12:30.672 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.303587_630.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.201196_756.1, shard=3 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.252013_419.1 2021-02-11T10:12:35.674 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:12:35.674 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:12:35.674 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:12:35.718 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:12:35.718 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:12:35.983 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4703921989677143155},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037887.303587_630.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.303587Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037887.201196_756.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.201196Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:28.881878Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037887.252013_419.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.252013Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:12:35.983 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.303587_630.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.201196_756.1, shard=3 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.252013_419.1 2021-02-11T10:12:40.984 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:12:40.985 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:12:40.985 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:12:41.027 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:12:41.027 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:12:41.293 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4703921989677143155},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037887.303587_630.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.303587Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037887.201196_756.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.201196Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:28.881878Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037887.252013_419.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.252013Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:12:41.293 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.303587_630.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.201196_756.1, shard=3 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.252013_419.1 2021-02-11T10:12:46.294 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:12:46.294 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:12:46.295 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:12:46.337 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:12:46.337 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:12:46.624 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4703921989677143155},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037887.303587_630.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.303587Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037887.201196_756.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.201196Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:28.881878Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037887.252013_419.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.252013Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:12:46.625 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.303587_630.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.201196_756.1, shard=3 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.252013_419.1 2021-02-11T10:12:51.625 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:12:51.626 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:12:51.626 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:12:51.669 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:12:51.669 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:12:51.944 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4703921989677143155},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1613037887.303587_630.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.303587Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1613037887.201196_756.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.201196Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"G00000000000000000001@00000000000000000000:00000000000000000778","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:05:28.881878Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1613037887.252013_419.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T10:04:47.252013Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T10:12:51.944 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.303587_630.1, shard=1 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.201196_756.1, shard=3 master=G00000000000000000001@00000000000000000000:00000000000000000000 target=1_1613037887.252013_419.1 2021-02-11T10:12:56.948 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_datalog_backing ... FAIL 2021-02-11T10:13:15.127 INFO:tasks.rgw_multisite_tests: 2021-02-11T10:13:15.127 INFO:tasks.rgw_multisite_tests:====================================================================== 2021-02-11T10:13:15.127 INFO:tasks.rgw_multisite_tests:FAIL: tasks.rgw_multi.tests.test_datalog_backing 2021-02-11T10:13:15.127 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2021-02-11T10:13:15.128 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2021-02-11T10:13:15.128 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2021-02-11T10:13:15.128 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2021-02-11T10:13:15.128 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5e5bb33c614118d912ca83c2e5f2be67b3acc04f/qa/tasks/rgw_multi/tests.py", line 1371, in test_datalog_backing 2021-02-11T10:13:15.128 INFO:tasks.rgw_multisite_tests: zonegroup_data_checkpoint(zonegroup_conns) 2021-02-11T10:13:15.128 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5e5bb33c614118d912ca83c2e5f2be67b3acc04f/qa/tasks/rgw_multi/tests.py", line 367, in zonegroup_data_checkpoint 2021-02-11T10:13:15.128 INFO:tasks.rgw_multisite_tests: zone_data_checkpoint(target_conn.zone, source_conn.zone) 2021-02-11T10:13:15.128 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_5e5bb33c614118d912ca83c2e5f2be67b3acc04f/qa/tasks/rgw_multi/tests.py", line 359, in zone_data_checkpoint 2021-02-11T10:13:15.128 INFO:tasks.rgw_multisite_tests: (target_zone.name, source_zone.name) 2021-02-11T10:13:15.128 INFO:tasks.rgw_multisite_tests:AssertionError: failed data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-02-11T10:13:15.128 INFO:tasks.rgw_multisite_tests: 2021-02-11T10:13:15.128 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2021-02-11T10:13:15.128 INFO:tasks.rgw_multisite_tests:Ran 29 tests in 1555.685s 2021-02-11T10:13:15.128 INFO:tasks.rgw_multisite_tests: 2021-02-11T10:13:15.129 INFO:tasks.rgw_multisite_tests:FAILED (failures=1) 2021-02-11T10:13:15.133 INFO:tasks.rgw_multisite_tests:log information for manual testing ... ok 2021-02-11T10:13:15.134 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:15.134 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:15.134 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:15.174 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:15.174 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:15.381 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 2021-02-11T10:13:15.423 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613037897.018955_662.1","last_update":"2021-02-11T10:04:57.018955Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:13:15.423 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T10:13:15.424 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:15.424 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:15.424 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:15.463 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:15.463 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:15.726 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037897.018955_662.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:04:57.018955Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:13:15.726 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T10:13:15.727 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T10:13:15.727 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:15.727 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:15.727 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:15.768 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:15.768 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:16.019 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037897.018955_662.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:04:57.018955Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:13:16.019 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T10:13:16.019 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:13:16.019 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:16.019 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:16.020 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:16.061 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:16.061 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:16.313 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037897.018955_662.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:04:57.018955Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:13:16.313 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:13:16.314 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:16.314 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:16.314 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:16.352 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:16.352 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:16.559 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 2021-02-11T10:13:16.599 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613037897.018955_662.1","last_update":"2021-02-11T10:04:57.018955Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:13:16.599 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:13:16.600 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:16.600 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:16.600 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:16.641 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:16.641 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:16.899 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613037897.018955_662.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:04:57.018955Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:13:16.900 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:13:16.968 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:16.968 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:16.969 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:17.006 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:17.007 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:17.196 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 2021-02-11T10:13:17.237 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613038396.963253_691.1","last_update":"2021-02-11T10:13:16.963253Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:13:17.237 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:13:17.238 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:17.238 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:17.238 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:17.279 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:17.279 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:17.509 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038396.963253_691.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:13:16.963253Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:13:17.509 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:13:17.632 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:17.632 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:17.632 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:17.672 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:17.672 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:17.883 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 2021-02-11T10:13:17.922 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613038396.963253_691.1","last_update":"2021-02-11T10:13:16.963253Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:13:17.923 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:13:17.923 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:17.923 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:17.923 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:17.961 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:17.961 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:18.209 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038396.963253_691.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:13:16.963253Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:13:18.209 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:13:18.365 INFO:tasks.rgw_multisite_tests:test low level implementation of s3 notifications ... ok 2021-02-11T10:13:18.366 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:18.366 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:18.366 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:18.403 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:18.403 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:18.603 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 2021-02-11T10:13:18.644 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613038398.351781_695.1","last_update":"2021-02-11T10:13:18.351781Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:13:18.645 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T10:13:18.645 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:18.645 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:18.645 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:18.686 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:18.686 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:18.954 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038398.351781_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:13:18.351781Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:13:18.954 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T10:13:18.955 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T10:13:18.955 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:18.955 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:18.955 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:18.995 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:18.995 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:19.243 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038398.351781_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:13:18.351781Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:13:19.243 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T10:13:19.243 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:13:19.243 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:19.243 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:19.243 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:19.282 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:19.282 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:19.522 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038398.351781_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:13:18.351781Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:13:19.523 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:13:19.523 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:19.523 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:19.523 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:19.563 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:19.564 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:19.778 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 2021-02-11T10:13:19.819 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613038398.351781_695.1","last_update":"2021-02-11T10:13:18.351781Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:13:19.820 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:13:19.820 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:19.820 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:19.820 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:19.861 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:19.861 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:20.109 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038398.351781_695.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:13:18.351781Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:13:20.110 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:13:20.156 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:20.157 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:20.157 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:20.193 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:20.194 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:20.385 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 2021-02-11T10:13:20.424 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613038400.151573_701.1","last_update":"2021-02-11T10:13:20.151573Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:13:20.425 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:13:20.425 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:20.425 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:20.425 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:20.468 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:20.468 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:20.722 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038400.151573_701.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:13:20.151573Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:13:20.723 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:13:20.763 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:20.763 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:20.763 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:20.805 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:20.805 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:21.026 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 2021-02-11T10:13:21.065 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613038400.151573_701.1","last_update":"2021-02-11T10:13:20.151573Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:13:21.065 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:13:21.065 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:21.066 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:21.066 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:21.107 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:21.107 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:21.346 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038400.151573_701.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:13:20.151573Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:13:21.347 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:13:21.764 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:13:21.764 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-75', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:13:21.764 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-75 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:13:21.819 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:21.820 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:22.048 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:13:22.046+0000 7f9417616080 1 bucket sync caught up with source: 2021-02-11T10:13:22.049 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, 00000000002.97.5, , 00000000001.108.5, 00000000001.55.5, 00000000001.107.5, , 00000000001.91.5, 00000000001.81.5, 00000000001.120.5, 00000000002.105.5] 2021-02-11T10:13:22.049 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, 00000000002.97.5, , 00000000001.108.5, 00000000001.55.5, 00000000001.107.5, , 00000000001.91.5, 00000000001.81.5, 00000000001.120.5, 00000000002.105.5] 2021-02-11T10:13:22.049 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:13:22.046+0000 7f9417616080 0 bucket checkpoint complete 2021-02-11T10:13:22.086 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2021-02-11T10:13:21.369928Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'tkscyf-75_notif', 'bucket': {'name': 'tkscyf-75', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::tkscyf-75', 'id': 'dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.26'}, 'object': {'key': '1', 'size': 0, 'etag': '9474006c', 'versionId': '', 'sequencer': '41032560DE45AA1A', 'metadata': [], 'tags': []}}, 'eventId': '1613038401.447366.9474006c', 'opaqueData': ''} 2021-02-11T10:13:22.086 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2021-02-11T10:13:21.352753Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'tkscyf-75_notif', 'bucket': {'name': 'tkscyf-75', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::tkscyf-75', 'id': 'dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.26'}, 'object': {'key': '0', 'size': 0, 'etag': 'de3aaca9', 'versionId': '', 'sequencer': '41032560E9ADBE1A', 'metadata': [], 'tags': []}}, 'eventId': '1613038401.448703.de3aaca9', 'opaqueData': ''} 2021-02-11T10:13:22.086 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2021-02-11T10:13:21.456094Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'tkscyf-75_notif', 'bucket': {'name': 'tkscyf-75', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::tkscyf-75', 'id': 'dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.26'}, 'object': {'key': '3', 'size': 0, 'etag': '2b04d467', 'versionId': '', 'sequencer': '41032560C977C926', 'metadata': [], 'tags': []}}, 'eventId': '1613038401.650737.2b04d467', 'opaqueData': ''} 2021-02-11T10:13:22.086 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2021-02-11T10:13:21.544213Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'tkscyf-75_notif', 'bucket': {'name': 'tkscyf-75', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::tkscyf-75', 'id': 'dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.26'}, 'object': {'key': '5', 'size': 0, 'etag': '0bd9d113', 'versionId': '', 'sequencer': '4103256047AFD426', 'metadata': [], 'tags': []}}, 'eventId': '1613038401.651472.0bd9d113', 'opaqueData': ''} 2021-02-11T10:13:22.086 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2021-02-11T10:13:21.500569Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'tkscyf-75_notif', 'bucket': {'name': 'tkscyf-75', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::tkscyf-75', 'id': 'dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.26'}, 'object': {'key': '4', 'size': 0, 'etag': 'b8397d1e', 'versionId': '', 'sequencer': '410325606C56B329', 'metadata': [], 'tags': []}}, 'eventId': '1613038401.699618.b8397d1e', 'opaqueData': ''} 2021-02-11T10:13:22.086 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2021-02-11T10:13:21.412166Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'tkscyf-75_notif', 'bucket': {'name': 'tkscyf-75', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::tkscyf-75', 'id': 'dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.26'}, 'object': {'key': '2', 'size': 0, 'etag': '722fbb40', 'versionId': '', 'sequencer': '4103256058B65F2B', 'metadata': [], 'tags': []}}, 'eventId': '1613038401.727692.722fbb40', 'opaqueData': ''} 2021-02-11T10:13:22.086 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2021-02-11T10:13:21.720407Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'tkscyf-75_notif', 'bucket': {'name': 'tkscyf-75', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::tkscyf-75', 'id': 'dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.26'}, 'object': {'key': '9', 'size': 0, 'etag': 'fdad4d3e', 'versionId': '', 'sequencer': '41032560FD976F2F', 'metadata': [], 'tags': []}}, 'eventId': '1613038401.795842.fdad4d3e', 'opaqueData': ''} 2021-02-11T10:13:22.087 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2021-02-11T10:13:21.676198Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'tkscyf-75_notif', 'bucket': {'name': 'tkscyf-75', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::tkscyf-75', 'id': 'dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.26'}, 'object': {'key': '8', 'size': 0, 'etag': 'da3abcdb', 'versionId': '', 'sequencer': '4103256063380E32', 'metadata': [], 'tags': []}}, 'eventId': '1613038401.839792.da3abcdb', 'opaqueData': ''} 2021-02-11T10:13:22.087 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2021-02-11T10:13:21.588478Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'tkscyf-75_notif', 'bucket': {'name': 'tkscyf-75', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::tkscyf-75', 'id': 'dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.26'}, 'object': {'key': '6', 'size': 0, 'etag': 'fd00d928', 'versionId': '', 'sequencer': '410325601B75F732', 'metadata': [], 'tags': []}}, 'eventId': '1613038401.855078.fd00d928', 'opaqueData': ''} 2021-02-11T10:13:22.087 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2021-02-11T10:13:21.632034Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'tkscyf-75_notif', 'bucket': {'name': 'tkscyf-75', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::tkscyf-75', 'id': 'dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.26'}, 'object': {'key': '7', 'size': 0, 'etag': 'f67c6c5a', 'versionId': '', 'sequencer': '41032560B7636335', 'metadata': [], 'tags': []}}, 'eventId': '1613038401.895706.f67c6c5a', 'opaqueData': ''} 2021-02-11T10:13:22.096 WARNING:tasks.rgw_multi.tests_ps:superfluous records are found 2021-02-11T10:13:22.286 INFO:tasks.rgw_multisite_tests:test s3 records fetching ... ok 2021-02-11T10:13:22.287 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:22.287 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:22.287 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:22.322 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:22.322 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:22.513 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 2021-02-11T10:13:22.543 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613038402.278578_705.1","last_update":"2021-02-11T10:13:22.278578Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:13:22.543 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T10:13:22.543 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:22.543 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:22.543 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:22.584 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:22.584 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:22.841 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038402.278578_705.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:13:22.278578Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:13:22.842 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T10:13:22.842 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T10:13:22.842 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:22.842 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:22.842 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:22.883 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:22.883 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:23.125 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038402.278578_705.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:13:22.278578Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:13:23.126 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T10:13:23.126 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:13:23.126 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:23.126 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:23.126 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:23.164 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:23.164 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:23.412 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038402.278578_705.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:13:22.278578Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:13:23.412 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:13:23.412 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:23.413 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:23.413 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:23.452 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:23.452 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:23.670 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 2021-02-11T10:13:23.711 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613038402.278578_705.1","last_update":"2021-02-11T10:13:22.278578Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:13:23.712 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:13:23.712 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:23.712 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:23.712 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:23.752 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:23.752 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:23.988 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038402.278578_705.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:13:22.278578Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:13:23.988 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:13:24.040 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:24.040 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:24.040 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:24.075 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:24.075 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:24.277 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 2021-02-11T10:13:24.307 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613038404.035396_712.1","last_update":"2021-02-11T10:13:24.035396Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:13:24.307 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:13:24.307 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:24.307 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:24.308 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:24.347 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:24.347 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:24.565 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038404.035396_712.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:13:24.035396Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:13:24.566 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:13:24.636 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:24.636 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:24.636 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:24.677 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:24.677 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:24.888 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 2021-02-11T10:13:24.927 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613038404.035396_712.1","last_update":"2021-02-11T10:13:24.035396Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:13:24.927 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:13:24.928 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:24.928 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:24.928 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:24.969 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:24.969 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:25.222 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038404.035396_712.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:13:24.035396Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:13:25.223 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:13:25.382 INFO:tasks.rgw_multisite_tests:test s3 notification set/get/delete ... ok 2021-02-11T10:13:25.383 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:25.383 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:25.383 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:25.422 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:25.422 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:25.625 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 2021-02-11T10:13:25.664 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613038405.370147_716.1","last_update":"2021-02-11T10:13:25.370147Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:13:25.664 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T10:13:25.664 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:25.664 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:25.665 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:25.706 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:25.706 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:25.970 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038405.370147_716.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:13:25.370147Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:13:25.970 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T10:13:25.971 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T10:13:25.971 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:25.971 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:25.971 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:26.012 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:26.012 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:26.267 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038405.370147_716.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:13:25.370147Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:13:26.267 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T10:13:26.268 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:13:26.268 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:26.268 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:26.268 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:26.306 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:26.306 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:26.549 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038405.370147_716.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:13:25.370147Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:13:26.549 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:13:26.549 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:26.550 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:26.550 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:26.587 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:26.587 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:26.795 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 2021-02-11T10:13:26.837 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613038405.370147_716.1","last_update":"2021-02-11T10:13:25.370147Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:13:26.837 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:13:26.837 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:26.837 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:26.837 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:26.878 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:26.878 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:27.137 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038405.370147_716.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:13:25.370147Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:13:27.137 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:13:27.345 INFO:tasks.rgw_multisite_tests:test s3 topics set/get/delete on master ... ok 2021-02-11T10:13:27.345 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:27.345 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:27.345 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:27.378 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:27.378 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:27.591 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 2021-02-11T10:13:27.631 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613038405.370147_716.1","last_update":"2021-02-11T10:13:25.370147Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:13:27.632 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T10:13:27.632 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:27.632 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:27.632 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:27.675 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:27.675 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:27.939 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038405.370147_716.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:13:25.370147Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:13:27.939 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T10:13:27.939 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T10:13:27.939 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:27.939 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:27.940 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:27.980 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:27.980 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:28.236 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038405.370147_716.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:13:25.370147Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:13:28.237 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T10:13:28.237 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:13:28.237 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:28.237 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:28.237 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:28.278 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:28.278 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:28.542 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038405.370147_716.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:13:25.370147Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:13:28.543 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:13:28.543 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:28.543 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:28.544 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:28.584 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:28.584 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:28.803 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 2021-02-11T10:13:28.843 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613038405.370147_716.1","last_update":"2021-02-11T10:13:25.370147Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:13:28.843 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:13:28.843 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:28.843 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:28.844 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:28.884 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:28.884 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:29.141 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038405.370147_716.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:13:25.370147Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:13:29.141 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:13:29.142 INFO:tasks.rgw_multisite_tests:test s3 topics with secret set/get/delete on master ... ok 2021-02-11T10:13:29.142 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:29.142 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:29.142 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:29.183 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:29.183 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:29.385 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 2021-02-11T10:13:29.422 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613038405.370147_716.1","last_update":"2021-02-11T10:13:25.370147Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:13:29.422 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T10:13:29.422 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:29.422 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:29.422 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:29.463 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:29.463 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:29.716 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038405.370147_716.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:13:25.370147Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:13:29.717 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T10:13:29.717 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T10:13:29.717 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:29.717 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:29.717 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:29.758 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:29.758 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:30.002 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038405.370147_716.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:13:25.370147Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:13:30.003 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T10:13:30.003 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:13:30.003 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:30.003 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:30.003 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:30.042 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:30.042 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:30.294 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038405.370147_716.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:13:25.370147Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:13:30.295 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:13:30.295 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:30.295 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:30.295 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:30.336 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:30.336 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:30.554 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 2021-02-11T10:13:30.595 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613038405.370147_716.1","last_update":"2021-02-11T10:13:25.370147Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:13:30.595 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:13:30.595 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:30.595 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:30.596 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:30.638 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:30.638 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:30.892 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038405.370147_716.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:13:25.370147Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:13:30.892 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:13:31.326 INFO:tasks.rgw_multisite_tests:test s3 notification set/get/delete on master ... ok 2021-02-11T10:13:31.327 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests_ps.test_ps_s3_notification_filter_on_master ... ok 2021-02-11T10:13:31.327 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests_ps.test_ps_s3_notification_filter ... ok 2021-02-11T10:13:31.327 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:31.327 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:31.327 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:31.366 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:31.367 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:31.565 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 2021-02-11T10:13:31.605 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613038411.268750_746.1","last_update":"2021-02-11T10:13:31.268750Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:13:31.605 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T10:13:31.605 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:31.605 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:31.605 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:31.647 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:31.647 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:31.917 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038411.268750_746.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:13:31.268750Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:13:31.918 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T10:13:31.918 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T10:13:31.918 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:31.918 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:31.918 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:31.959 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:31.959 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:32.206 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038411.268750_746.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:13:31.268750Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:13:32.386 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T10:13:32.386 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:13:32.386 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:32.387 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:32.387 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:32.428 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:32.428 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:32.691 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038411.268750_746.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:13:31.268750Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:13:32.691 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:13:32.691 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:32.691 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:32.692 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:32.730 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:32.730 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:32.938 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 2021-02-11T10:13:32.981 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613038411.268750_746.1","last_update":"2021-02-11T10:13:31.268750Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:13:32.982 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:13:32.982 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:32.982 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:32.982 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:33.023 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:33.023 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:33.274 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038411.268750_746.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:13:31.268750Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:13:33.275 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:13:33.523 INFO:tasks.rgw_multisite_tests:test s3 notification set/get/delete on master ... ok 2021-02-11T10:13:33.523 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests_ps.test_objcet_timing ... ok 2021-02-11T10:13:33.524 INFO:tasks.rgw_multisite_tests:test pushing amqp s3 notification on master ... ok 2021-02-11T10:13:33.524 INFO:tasks.rgw_multisite_tests:test reservation cleanup after gateway crash ... ok 2021-02-11T10:13:33.524 INFO:tasks.rgw_multisite_tests:test gateway recovery of persistent notifications ... ok 2021-02-11T10:13:33.524 INFO:tasks.rgw_multisite_tests:test pushing persistent notification via two gateways ... ok 2021-02-11T10:13:33.524 INFO:tasks.rgw_multisite_tests:test pushing persistent notification when one of the endpoints has error ... ok 2021-02-11T10:13:33.525 INFO:tasks.rgw_multisite_tests:test pushing persistent notification http ... ok 2021-02-11T10:13:33.525 INFO:tasks.rgw_multisite_tests:test pushing persistent notification amqp ... ok 2021-02-11T10:13:33.525 INFO:tasks.rgw_multisite_tests:test pushing persistent notification of large notifications ... ok 2021-02-11T10:13:33.525 INFO:tasks.rgw_multisite_tests:test pushing persistent notification pushback ... ok 2021-02-11T10:13:33.525 INFO:tasks.rgw_multisite_tests:test pushing kafka s3 notification on master ... ok 2021-02-11T10:13:33.525 INFO:tasks.rgw_multisite_tests:test pushing kafka s3 notification on master ... ok 2021-02-11T10:13:33.526 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests_ps.test_ps_s3_notification_push_kafka_security_ssl ... ok 2021-02-11T10:13:33.526 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests_ps.test_ps_s3_notification_push_kafka_security_ssl_sasl ... ok 2021-02-11T10:13:33.526 INFO:tasks.rgw_multisite_tests:test deletion of multiple keys on master ... ok 2021-02-11T10:13:33.526 INFO:tasks.rgw_multisite_tests:test pushing http s3 notification on master ... ok 2021-02-11T10:13:33.526 INFO:tasks.rgw_multisite_tests:test that opaque id set in topic, is sent in notification ... ok 2021-02-11T10:13:33.527 INFO:tasks.rgw_multisite_tests:test that opaque id set in topic, is sent in notification on master ... ok 2021-02-11T10:13:33.527 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:33.527 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:33.527 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:33.562 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:33.562 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:33.769 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 2021-02-11T10:13:33.812 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613038413.513875_764.1","last_update":"2021-02-11T10:13:33.513875Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:13:33.813 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T10:13:33.813 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:33.813 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:33.813 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:33.853 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:33.853 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:34.108 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038413.513875_764.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:13:33.513875Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:13:34.108 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T10:13:34.108 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T10:13:34.109 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:34.109 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:34.109 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:34.150 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:34.150 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:34.405 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038413.513875_764.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:13:33.513875Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:13:34.405 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T10:13:34.406 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:13:34.406 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:34.406 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:34.406 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:34.450 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:34.450 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:34.699 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038413.513875_764.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:13:33.513875Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:13:34.699 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:13:34.699 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:34.699 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:34.699 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:34.738 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:34.738 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:34.921 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 2021-02-11T10:13:34.955 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613038413.513875_764.1","last_update":"2021-02-11T10:13:33.513875Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:13:34.955 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:13:34.955 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:34.956 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:34.956 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:34.998 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:34.998 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:35.254 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038413.513875_764.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:13:33.513875Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:13:35.254 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:13:35.276 INFO:tasks.rgw_multisite_tests:test set/get/delete of topic ... ok 2021-02-11T10:13:35.276 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:35.277 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:35.277 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:35.324 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:35.324 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:35.520 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 2021-02-11T10:13:35.561 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613038413.513875_764.1","last_update":"2021-02-11T10:13:33.513875Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:13:35.561 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T10:13:35.561 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:35.561 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:35.562 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:35.605 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:35.606 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:35.883 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038413.513875_764.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:13:33.513875Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:13:35.883 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T10:13:35.883 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T10:13:35.883 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:35.883 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:35.884 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:35.924 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:35.924 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:36.170 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038413.513875_764.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:13:33.513875Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:13:36.171 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T10:13:36.171 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:13:36.171 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:36.171 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:36.171 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:36.213 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:36.213 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:36.477 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038413.513875_764.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:13:33.513875Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:13:36.477 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:13:36.477 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:36.477 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:36.478 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:36.517 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:36.517 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:36.720 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 2021-02-11T10:13:36.758 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613038413.513875_764.1","last_update":"2021-02-11T10:13:33.513875Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:13:36.759 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:13:36.759 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:36.759 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:36.759 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:36.799 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:36.799 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:37.047 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038413.513875_764.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:13:33.513875Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:13:37.047 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:13:37.065 INFO:tasks.rgw_multisite_tests:test set topic with endpoint ... ok 2021-02-11T10:13:37.066 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:37.066 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:37.066 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:37.106 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:37.106 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:37.308 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 2021-02-11T10:13:37.342 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613038413.513875_764.1","last_update":"2021-02-11T10:13:33.513875Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:13:37.342 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T10:13:37.342 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:37.343 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:37.343 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:37.387 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:37.388 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:37.638 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038413.513875_764.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:13:33.513875Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:13:37.638 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T10:13:37.638 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T10:13:37.638 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:37.639 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:37.639 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:37.680 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:37.680 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:37.921 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038413.513875_764.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:13:33.513875Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:13:37.921 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T10:13:37.921 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:13:37.921 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:37.921 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:37.922 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:37.963 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:37.963 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:38.207 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038413.513875_764.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:13:33.513875Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:13:38.207 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:13:38.207 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:38.207 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:38.207 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:38.246 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:38.246 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:38.439 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 2021-02-11T10:13:38.478 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613038413.513875_764.1","last_update":"2021-02-11T10:13:33.513875Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:13:38.478 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:13:38.478 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:38.478 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:38.478 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:38.519 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:38.519 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:38.766 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038413.513875_764.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:13:33.513875Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:13:38.767 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:13:38.827 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:38.828 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:38.828 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:38.860 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:38.860 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:39.053 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 2021-02-11T10:13:39.091 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613038418.821126_774.1","last_update":"2021-02-11T10:13:38.821126Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:13:39.092 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:13:39.092 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:39.092 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:39.092 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:39.128 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:39.128 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:39.370 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038418.821126_774.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:13:38.821126Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:13:39.370 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:13:39.506 INFO:tasks.rgw_multisite_tests:test set/get/delete of notification ... ok 2021-02-11T10:13:39.506 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:39.506 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:39.506 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:39.543 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:39.543 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:39.747 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 2021-02-11T10:13:39.787 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613038419.492948_777.1","last_update":"2021-02-11T10:13:39.492948Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:13:39.787 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T10:13:39.787 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:39.787 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:39.788 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:39.828 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:39.828 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:40.084 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038419.492948_777.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:13:39.492948Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:13:40.084 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T10:13:40.084 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T10:13:40.084 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:40.084 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:40.085 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:40.125 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:40.125 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:40.370 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038419.492948_777.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:13:39.492948Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:13:40.370 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T10:13:40.370 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:13:40.370 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:40.370 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:40.370 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:40.406 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:40.406 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:40.628 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038419.492948_777.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:13:39.492948Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:13:40.628 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:13:40.629 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:40.629 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:40.629 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:40.667 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:40.667 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:40.878 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 2021-02-11T10:13:40.919 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613038419.492948_777.1","last_update":"2021-02-11T10:13:39.492948Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:13:40.919 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:13:40.919 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:40.919 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:40.920 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:40.962 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:40.962 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:41.212 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038419.492948_777.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:13:39.492948Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:13:41.212 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:13:41.268 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:41.268 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:41.268 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:41.303 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:41.303 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:41.499 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 2021-02-11T10:13:41.531 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613038421.263292_783.1","last_update":"2021-02-11T10:13:41.263292Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:13:41.531 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:13:41.531 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:41.531 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:41.532 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:41.568 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:41.568 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:41.809 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038421.263292_783.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:13:41.263292Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:13:41.810 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:13:41.947 INFO:tasks.rgw_multisite_tests:test set/get/delete of notification on specific events ... ok 2021-02-11T10:13:41.948 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:41.948 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:41.948 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:41.985 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:41.985 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:42.183 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 2021-02-11T10:13:42.219 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613038421.934748_786.1","last_update":"2021-02-11T10:13:41.934748Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:13:42.219 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T10:13:42.219 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:42.219 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:42.220 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:42.260 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:42.260 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:42.530 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038421.934748_786.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:13:41.934748Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:13:42.530 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T10:13:42.530 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T10:13:42.531 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:42.531 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:42.531 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:42.571 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:42.571 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:42.873 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038421.934748_786.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:13:41.934748Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:13:42.873 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T10:13:42.873 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:13:42.874 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:42.874 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:42.874 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:42.913 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:42.913 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:43.172 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038421.934748_786.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:13:41.934748Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:13:43.172 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:13:43.173 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:43.173 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:43.173 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:43.211 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:43.211 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:43.391 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 2021-02-11T10:13:43.427 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613038421.934748_786.1","last_update":"2021-02-11T10:13:41.934748Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:13:43.427 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:13:43.428 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:43.428 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:43.428 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:43.466 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:43.466 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:43.714 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038421.934748_786.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:13:41.934748Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:13:43.714 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:13:43.774 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:43.774 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:43.775 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:43.809 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:43.809 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:44.002 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 2021-02-11T10:13:44.045 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613038423.768332_792.1","last_update":"2021-02-11T10:13:43.768332Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:13:44.045 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:13:44.045 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:44.045 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:44.045 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:44.087 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:44.087 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:44.336 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038423.768332_792.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:13:43.768332Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:13:44.336 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:13:44.778 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-84', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:13:44.779 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-84', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:13:44.779 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-84 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:13:44.837 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:44.837 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:45.088 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:13:45.081+0000 7f7708bf9080 1 waiting for incremental sync to catch up: 2021-02-11T10:13:45.088 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, 00000000002.95.5, , 00000000001.89.5, 00000000001.113.5, 00000000001.91.5, , , 00000000001.112.5, 00000000001.115.5, 00000000002.83.5] 2021-02-11T10:13:45.088 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, 00000000002.95.5, , 00000000001.89.5, 00000000001.113.5, 00000000001.91.5, , 00000000001.112.5, 00000000001.112.5, 00000000001.115.5, 00000000002.83.5] 2021-02-11T10:13:50.091 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:13:50.085+0000 7f7708bf9080 1 waiting for incremental sync to catch up: 2021-02-11T10:13:50.091 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, 00000000002.95.5, , 00000000001.89.5, 00000000001.113.5, 00000000001.91.5, , , 00000000001.112.5, 00000000001.115.5, 00000000002.83.5] 2021-02-11T10:13:50.091 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, 00000000002.95.5, , 00000000001.89.5, 00000000001.113.5, 00000000001.91.5, , 00000000001.112.5, 00000000001.112.5, 00000000001.115.5, 00000000002.83.5] 2021-02-11T10:13:55.095 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:13:55.089+0000 7f7708bf9080 1 bucket sync caught up with source: 2021-02-11T10:13:55.095 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, 00000000002.95.5, , 00000000001.89.5, 00000000001.113.5, 00000000001.91.5, , 00000000001.112.5, 00000000001.112.5, 00000000001.115.5, 00000000002.83.5] 2021-02-11T10:13:55.095 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, 00000000002.95.5, , 00000000001.89.5, 00000000001.113.5, 00000000001.91.5, , 00000000001.112.5, 00000000001.112.5, 00000000001.115.5, 00000000002.83.5] 2021-02-11T10:13:55.095 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:13:55.089+0000 7f7708bf9080 0 bucket checkpoint complete 2021-02-11T10:13:55.140 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "0" type: "OBJECT_CREATE" 2021-02-11T10:13:55.140 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "1" type: "OBJECT_CREATE" 2021-02-11T10:13:55.140 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "5" type: "OBJECT_CREATE" 2021-02-11T10:13:55.140 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "9" type: "OBJECT_CREATE" 2021-02-11T10:13:55.140 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "6" type: "OBJECT_CREATE" 2021-02-11T10:13:55.140 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "3" type: "OBJECT_CREATE" 2021-02-11T10:13:55.140 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "7" type: "OBJECT_CREATE" 2021-02-11T10:13:55.140 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "4" type: "OBJECT_CREATE" 2021-02-11T10:13:55.140 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "8" type: "OBJECT_CREATE" 2021-02-11T10:13:55.140 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "2" type: "OBJECT_CREATE" 2021-02-11T10:13:55.159 DEBUG:tasks.rgw_multi.tests_ps:superfluous events are found 2021-02-11T10:13:55.250 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-84', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:13:55.250 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-84', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:13:55.250 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-84 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:13:55.297 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:55.297 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:55.520 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:13:55.517+0000 7fca2617e080 1 bucket sync caught up with source: 2021-02-11T10:13:55.521 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, 00000000004.99.5, , 00000000002.93.5, 00000000002.115.5, 00000000002.95.5, , 00000000002.114.5, 00000000002.114.5, 00000000002.117.5, 00000000004.87.5] 2021-02-11T10:13:55.521 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, 00000000004.99.5, , 00000000002.93.5, 00000000002.115.5, 00000000002.95.5, , 00000000002.114.5, 00000000002.114.5, 00000000002.117.5, 00000000004.87.5] 2021-02-11T10:13:55.521 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:13:55.517+0000 7fca2617e080 0 bucket checkpoint complete 2021-02-11T10:13:55.669 INFO:tasks.rgw_multisite_tests:test set/get/delete of subscription ... ok 2021-02-11T10:13:55.669 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:55.669 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:55.669 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:55.706 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:55.706 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:55.877 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 2021-02-11T10:13:55.912 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613038435.660028_799.1","last_update":"2021-02-11T10:13:55.660028Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:13:55.912 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T10:13:55.913 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:55.913 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:55.913 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:55.953 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:55.954 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:56.201 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038435.660028_799.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:13:55.660028Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:13:56.202 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T10:13:56.202 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T10:13:56.202 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:56.202 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:56.202 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:56.243 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:56.243 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:56.486 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038435.660028_799.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:13:55.660028Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:13:56.486 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T10:13:56.486 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:13:56.486 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:56.486 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:56.487 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:56.526 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:56.526 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:56.769 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038435.660028_799.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:13:55.660028Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:13:56.769 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:13:56.769 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:56.769 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:56.769 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:56.809 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:56.809 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:57.020 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 2021-02-11T10:13:57.059 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613038435.660028_799.1","last_update":"2021-02-11T10:13:55.660028Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:13:57.060 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:13:57.060 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:57.060 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:57.060 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:57.100 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:57.101 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:57.344 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038435.660028_799.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:13:55.660028Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:13:57.345 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:13:57.355 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['topic', 'list', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:13:57.355 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'topic', 'list', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:13:57.356 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 topic list --uid rgw-multisite-test-user --rgw-zone test-zone4 --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T10:13:57.423 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:57.423 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:57.662 INFO:teuthology.orchestra.run.smithi098.stdout:{"topics":[{"topic":{"user":"","name":"tkscyf-85_topic","dest":{"bucket_name":"","oid_prefix":"","push_endpoint":"","push_endpoint_args":"","push_endpoint_topic":"tkscyf-85_topic","stored_secret":"false","persistent":"false"},"arn":"arn:aws:sns:test-zonegroup::tkscyf-85_topic","opaqueData":""},"subs":[]}]} 2021-02-11T10:13:57.662 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['topic', 'get', '--uid', 'rgw-multisite-test-user', '--topic', 'tkscyf-85_topic', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:13:57.662 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'topic', 'get', '--uid', 'rgw-multisite-test-user', '--topic', 'tkscyf-85_topic', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:13:57.663 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 topic get --uid rgw-multisite-test-user --topic tkscyf-85_topic --rgw-zone test-zone4 --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T10:13:57.744 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:57.744 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:58.012 INFO:teuthology.orchestra.run.smithi098.stdout:{"topic":{"user":"","name":"tkscyf-85_topic","dest":{"bucket_name":"","oid_prefix":"","push_endpoint":"","push_endpoint_args":"","push_endpoint_topic":"tkscyf-85_topic","stored_secret":"false","persistent":"false"},"arn":"arn:aws:sns:test-zonegroup::tkscyf-85_topic","opaqueData":""},"subs":[]} 2021-02-11T10:13:58.035 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['topic', 'list', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:13:58.035 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'topic', 'list', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:13:58.035 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 topic list --uid rgw-multisite-test-user --rgw-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T10:13:58.075 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:58.076 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:58.322 INFO:teuthology.orchestra.run.smithi094.stdout:{"topics":[{"topic":{"user":"","name":"tkscyf-85_topic","dest":{"bucket_name":"","oid_prefix":"","push_endpoint":"amqp://127.0.0.1:7001/vhost_1","push_endpoint_args":"Attributes.entry.1.key=push-endpoint&Attributes.entry.1.value=amqp://127.0.0.1:7001/vhost_1&Attributes.entry.2.key=amqp-exchange&Attributes.entry.2.value=amqp.direct&Attributes.entry.3.key=amqp-ack-level&Attributes.entry.3.value=none&Version=2010-03-31&amqp-ack-level=none&amqp-exchange=amqp.direct&push-endpoint=amqp://127.0.0.1:7001/vhost_1","push_endpoint_topic":"tkscyf-85_topic","stored_secret":"false","persistent":"false"},"arn":"arn:aws:sns:test-zonegroup::tkscyf-85_topic","opaqueData":""},"subs":[]}]} 2021-02-11T10:13:58.322 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['topic', 'get', '--uid', 'rgw-multisite-test-user', '--topic', 'tkscyf-85_topic', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:13:58.323 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'topic', 'get', '--uid', 'rgw-multisite-test-user', '--topic', 'tkscyf-85_topic', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:13:58.323 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 topic get --uid rgw-multisite-test-user --topic tkscyf-85_topic --rgw-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T10:13:58.403 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:58.403 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:58.655 INFO:teuthology.orchestra.run.smithi094.stdout:{"topic":{"user":"","name":"tkscyf-85_topic","dest":{"bucket_name":"","oid_prefix":"","push_endpoint":"amqp://127.0.0.1:7001/vhost_1","push_endpoint_args":"Attributes.entry.1.key=push-endpoint&Attributes.entry.1.value=amqp://127.0.0.1:7001/vhost_1&Attributes.entry.2.key=amqp-exchange&Attributes.entry.2.value=amqp.direct&Attributes.entry.3.key=amqp-ack-level&Attributes.entry.3.value=none&Version=2010-03-31&amqp-ack-level=none&amqp-exchange=amqp.direct&push-endpoint=amqp://127.0.0.1:7001/vhost_1","push_endpoint_topic":"tkscyf-85_topic","stored_secret":"false","persistent":"false"},"arn":"arn:aws:sns:test-zonegroup::tkscyf-85_topic","opaqueData":""},"subs":[]} 2021-02-11T10:13:58.701 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:58.702 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:58.702 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:58.739 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:58.739 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:58.923 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 2021-02-11T10:13:58.957 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613038438.696036_809.1","last_update":"2021-02-11T10:13:58.696036Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:13:58.957 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:13:58.958 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:58.958 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:13:58.958 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:13:58.997 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:58.997 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:59.260 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038438.696036_809.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:13:58.696036Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:13:59.260 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:13:59.282 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['subscription', 'get', '--uid', 'rgw-multisite-test-user', '--subscription', 'tkscyf-85_sub', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:13:59.282 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'subscription', 'get', '--uid', 'rgw-multisite-test-user', '--subscription', 'tkscyf-85_sub', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:13:59.283 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 subscription get --uid rgw-multisite-test-user --subscription tkscyf-85_sub --rgw-zone test-zone4 --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T10:13:59.344 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:13:59.344 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:13:59.591 INFO:teuthology.orchestra.run.smithi098.stdout:{"user":"","name":"tkscyf-85_sub","topic":"tkscyf-85_topic","dest":{"bucket_name":"pubsub-rgw-multisite-test-user-tkscyf-85_topic","oid_prefix":"tkscyf-85_sub/","push_endpoint":"","push_endpoint_args":"topic=tkscyf-85_topic","push_endpoint_topic":"tkscyf-85_topic","stored_secret":"false","persistent":"false"},"s3_id":""} 2021-02-11T10:14:04.451 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-85', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:14:04.451 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-85', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:14:04.451 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-85 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:14:04.500 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:04.500 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:04.715 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:14:04.709+0000 7f5f8a1ad080 1 bucket sync caught up with source: 2021-02-11T10:14:04.716 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [00000000011.98.5, 00000000011.118.5, 00000000010.114.5, 00000000010.68.5, 00000000009.68.5, 00000000007.126.5, 00000000010.133.5, 00000000010.148.5, 00000000011.95.5, 00000000011.116.5, 00000000010.101.5] 2021-02-11T10:14:04.716 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [00000000011.98.5, 00000000011.118.5, 00000000010.114.5, 00000000010.68.5, 00000000009.68.5, 00000000007.126.5, 00000000010.133.5, 00000000010.148.5, 00000000011.95.5, 00000000011.116.5, 00000000010.101.5] 2021-02-11T10:14:04.716 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:14:04.709+0000 7f5f8a1ad080 0 bucket checkpoint complete 2021-02-11T10:14:04.747 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['subscription', 'pull', '--uid', 'rgw-multisite-test-user', '--subscription', 'tkscyf-85_sub', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:14:04.747 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'subscription', 'pull', '--uid', 'rgw-multisite-test-user', '--subscription', 'tkscyf-85_sub', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:14:04.747 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 subscription pull --uid rgw-multisite-test-user --subscription tkscyf-85_sub --rgw-zone test-zone4 --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T10:14:04.785 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:04.785 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:05.041 INFO:teuthology.orchestra.run.smithi098.stdout:{"next_marker":"tkscyf-85_sub/1613038444.024211.4d86bbe4","is_truncated":"true","events":[{"id":"1613038439.876245.2cc931d5","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:13:59.876245Z","info":{"attrs":{"mtime":"2021-02-11T10:13:59.656612Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"2"}}},{"id":"1613038439.880739.ba939252","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:13:59.880739Z","info":{"attrs":{"mtime":"2021-02-11T10:13:59.612964Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"1"}}},{"id":"1613038439.881298.5959feb9","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:13:59.881298Z","info":{"attrs":{"mtime":"2021-02-11T10:13:59.700589Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"3"}}},{"id":"1613038439.926202.c8e546c3","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:13:59.926202Z","info":{"attrs":{"mtime":"2021-02-11T10:13:59.788579Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"5"}}},{"id":"1613038439.927447.3a1976d0","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:13:59.927447Z","info":{"attrs":{"mtime":"2021-02-11T10:13:59.594973Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"0"}}},{"id":"1613038439.975816.872bd973","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:13:59.975816Z","info":{"attrs":{"mtime":"2021-02-11T10:13:59.744684Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"4"}}},{"id":"1613038440.063224.9616adf6","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:00.063224Z","info":{"attrs":{"mtime":"2021-02-11T10:13:59.876219Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"7"}}},{"id":"1613038440.063503.94a3b28c","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:00.063503Z","info":{"attrs":{"mtime":"2021-02-11T10:13:59.832240Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"6"}}},{"id":"1613038440.143669.22dc6b16","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:00.143669Z","info":{"attrs":{"mtime":"2021-02-11T10:13:59.921094Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"8"}}},{"id":"1613038440.217687.93efed7a","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:00.217687Z","info":{"attrs":{"mtime":"2021-02-11T10:14:00.152136Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"13"}}},{"id":"1613038440.217903.17c560f8","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:00.217903Z","info":{"attrs":{"mtime":"2021-02-11T10:14:00.056279Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"11"}}},{"id":"1613038440.219516.a8e32462","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:00.219516Z","info":{"attrs":{"mtime":"2021-02-11T10:14:00.104155Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"12"}}},{"id":"1613038440.310329.40434a3f","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:00.310329Z","info":{"attrs":{"mtime":"2021-02-11T10:14:00.008402Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"10"}}},{"id":"1613038440.414485.4fb45978","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:00.414485Z","info":{"attrs":{"mtime":"2021-02-11T10:13:59.964309Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"9"}}},{"id":"1613038440.415891.5d289f60","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:00.415891Z","info":{"attrs":{"mtime":"2021-02-11T10:14:00.196093Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"14"}}},{"id":"1613038440.418990.dbdfc81a","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:00.418990Z","info":{"attrs":{"mtime":"2021-02-11T10:14:00.376612Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"18"}}},{"id":"1613038440.457938.3ba2e853","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:00.457938Z","info":{"attrs":{"mtime":"2021-02-11T10:14:00.328547Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"17"}}},{"id":"1613038440.458249.04786e5c","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:00.458249Z","info":{"attrs":{"mtime":"2021-02-11T10:14:00.240064Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"15"}}},{"id":"1613038440.459851.35e73d9f","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:00.459851Z","info":{"attrs":{"mtime":"2021-02-11T10:14:00.284059Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"16"}}},{"id":"1613038440.619002.40eb98d6","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:00.619002Z","info":{"attrs":{"mtime":"2021-02-11T10:14:00.420120Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"19"}}},{"id":"1613038440.620796.940062b2","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:00.620796Z","info":{"attrs":{"mtime":"2021-02-11T10:14:00.556273Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"22"}}},{"id":"1613038440.667779.944d1163","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:00.667779Z","info":{"attrs":{"mtime":"2021-02-11T10:14:00.464323Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"20"}}},{"id":"1613038440.699978.aa82a4c9","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:00.699978Z","info":{"attrs":{"mtime":"2021-02-11T10:14:00.508133Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"21"}}},{"id":"1613038440.819416.2ce28479","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:00.819416Z","info":{"attrs":{"mtime":"2021-02-11T10:14:00.776113Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"27"}}},{"id":"1613038440.820034.906b4eee","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:00.820034Z","info":{"attrs":{"mtime":"2021-02-11T10:14:00.600681Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"23"}}},{"id":"1613038440.903775.37c0449d","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:00.903775Z","info":{"attrs":{"mtime":"2021-02-11T10:14:00.732173Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"26"}}},{"id":"1613038440.903931.e97aa2c3","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:00.903931Z","info":{"attrs":{"mtime":"2021-02-11T10:14:00.688213Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"25"}}},{"id":"1613038440.951807.efe2c952","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:00.951807Z","info":{"attrs":{"mtime":"2021-02-11T10:14:00.644336Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"24"}}},{"id":"1613038440.951988.1d86f773","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:00.951988Z","info":{"attrs":{"mtime":"2021-02-11T10:14:00.820193Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"28"}}},{"id":"1613038441.020806.63d382b3","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:01.020806Z","info":{"attrs":{"mtime":"2021-02-11T10:14:00.908113Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"30"}}},{"id":"1613038441.024073.d40d1a92","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:01.024073Z","info":{"attrs":{"mtime":"2021-02-11T10:14:00.864262Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"29"}}},{"id":"1613038441.099712.64a2a027","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:01.099712Z","info":{"attrs":{"mtime":"2021-02-11T10:14:00.952224Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"31"}}},{"id":"1613038441.103680.38091638","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:01.103680Z","info":{"attrs":{"mtime":"2021-02-11T10:14:00.996169Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"32"}}},{"id":"1613038441.220271.956ddc7b","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:01.220271Z","info":{"attrs":{"mtime":"2021-02-11T10:14:01.044147Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"33"}}},{"id":"1613038441.262693.9cbf9889","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:01.262693Z","info":{"attrs":{"mtime":"2021-02-11T10:14:01.088240Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"34"}}},{"id":"1613038441.265632.5858367c","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:01.265632Z","info":{"attrs":{"mtime":"2021-02-11T10:14:01.132457Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"35"}}},{"id":"1613038441.311898.19dfd7ab","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:01.311898Z","info":{"attrs":{"mtime":"2021-02-11T10:14:01.176669Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"36"}}},{"id":"1613038441.419983.dd76bb47","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:01.419983Z","info":{"attrs":{"mtime":"2021-02-11T10:14:01.308397Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"39"}}},{"id":"1613038441.427665.d6d4b95f","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:01.427665Z","info":{"attrs":{"mtime":"2021-02-11T10:14:01.396363Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"41"}}},{"id":"1613038441.463642.ec0a042a","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:01.463642Z","info":{"attrs":{"mtime":"2021-02-11T10:14:01.264337Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"38"}}},{"id":"1613038441.503879.8eba1ca1","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:01.503879Z","info":{"attrs":{"mtime":"2021-02-11T10:14:01.220514Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"37"}}},{"id":"1613038441.505957.0b1d0d4b","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:01.505957Z","info":{"attrs":{"mtime":"2021-02-11T10:14:01.352401Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"40"}}},{"id":"1613038441.618908.b30b53e9","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:01.618908Z","info":{"attrs":{"mtime":"2021-02-11T10:14:01.440123Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"42"}}},{"id":"1613038441.619109.3058baa0","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:01.619109Z","info":{"attrs":{"mtime":"2021-02-11T10:14:01.572142Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"45"}}},{"id":"1613038441.658551.730bdb32","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:01.658551Z","info":{"attrs":{"mtime":"2021-02-11T10:14:01.528221Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"44"}}},{"id":"1613038441.658695.c5034859","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:01.658695Z","info":{"attrs":{"mtime":"2021-02-11T10:14:01.484463Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"43"}}},{"id":"1613038441.819497.e0763412","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:01.819497Z","info":{"attrs":{"mtime":"2021-02-11T10:14:01.616225Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"46"}}},{"id":"1613038441.820489.bdbc20be","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:01.820489Z","info":{"attrs":{"mtime":"2021-02-11T10:14:01.792319Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"50"}}},{"id":"1613038441.825699.92647182","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:01.825699Z","info":{"attrs":{"mtime":"2021-02-11T10:14:01.704054Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"48"}}},{"id":"1613038441.863738.6d612a2c","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:01.863738Z","info":{"attrs":{"mtime":"2021-02-11T10:14:01.748126Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"49"}}},{"id":"1613038441.899637.007f64fe","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:01.899637Z","info":{"attrs":{"mtime":"2021-02-11T10:14:01.660206Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"47"}}},{"id":"1613038442.022480.b1d1d3d2","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:02.022480Z","info":{"attrs":{"mtime":"2021-02-11T10:14:01.924441Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"53"}}},{"id":"1613038442.063316.6812ce94","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:02.063316Z","info":{"attrs":{"mtime":"2021-02-11T10:14:01.968404Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"54"}}},{"id":"1613038442.063832.4ca24ad6","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:02.063832Z","info":{"attrs":{"mtime":"2021-02-11T10:14:01.880077Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"52"}}},{"id":"1613038442.321325.1b81da26","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:02.321325Z","info":{"attrs":{"mtime":"2021-02-11T10:14:02.012702Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"55"}}},{"id":"1613038442.323754.aba8a855","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:02.323754Z","info":{"attrs":{"mtime":"2021-02-11T10:14:02.188598Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"59"}}},{"id":"1613038442.324270.e54c6e13","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:02.324270Z","info":{"attrs":{"mtime":"2021-02-11T10:14:01.836344Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"51"}}},{"id":"1613038442.325297.85d0fed3","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:02.325297Z","info":{"attrs":{"mtime":"2021-02-11T10:14:02.144672Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"58"}}},{"id":"1613038442.326167.0520d947","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:02.326167Z","info":{"attrs":{"mtime":"2021-02-11T10:14:02.100944Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"57"}}},{"id":"1613038442.327910.18b0cdea","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:02.327910Z","info":{"attrs":{"mtime":"2021-02-11T10:14:02.056350Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"56"}}},{"id":"1613038442.430567.96ced051","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:02.430567Z","info":{"attrs":{"mtime":"2021-02-11T10:14:02.368798Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"63"}}},{"id":"1613038442.430863.5c1cd7ef","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:02.430863Z","info":{"attrs":{"mtime":"2021-02-11T10:14:02.236363Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"60"}}},{"id":"1613038442.516032.ce49e965","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:02.516032Z","info":{"attrs":{"mtime":"2021-02-11T10:14:02.324181Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"62"}}},{"id":"1613038442.618924.fb5a91fa","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:02.618924Z","info":{"attrs":{"mtime":"2021-02-11T10:14:02.588493Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"68"}}},{"id":"1613038442.659874.2b767c21","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:02.659874Z","info":{"attrs":{"mtime":"2021-02-11T10:14:02.456096Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"65"}}},{"id":"1613038442.660866.902e8cdf","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:02.660866Z","info":{"attrs":{"mtime":"2021-02-11T10:14:02.414046Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"64"}}},{"id":"1613038442.661067.f018bccd","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:02.661067Z","info":{"attrs":{"mtime":"2021-02-11T10:14:02.544137Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"67"}}},{"id":"1613038442.662396.fc1a4f79","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:02.662396Z","info":{"attrs":{"mtime":"2021-02-11T10:14:02.280252Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"61"}}},{"id":"1613038442.665323.4de5833d","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:02.665323Z","info":{"attrs":{"mtime":"2021-02-11T10:14:02.500016Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"66"}}},{"id":"1613038442.824383.3a667bb5","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:02.824383Z","info":{"attrs":{"mtime":"2021-02-11T10:14:02.632080Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"69"}}},{"id":"1613038442.824882.2aeedc4b","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:02.824882Z","info":{"attrs":{"mtime":"2021-02-11T10:14:02.764279Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"72"}}},{"id":"1613038442.828154.bd5d3d65","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:02.828154Z","info":{"attrs":{"mtime":"2021-02-11T10:14:02.720405Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"71"}}},{"id":"1613038442.828455.bb034400","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:02.828455Z","info":{"attrs":{"mtime":"2021-02-11T10:14:02.676141Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"70"}}},{"id":"1613038443.027656.ca6ca02a","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:03.027656Z","info":{"attrs":{"mtime":"2021-02-11T10:14:02.851956Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"74"}}},{"id":"1613038443.030280.0caae0c3","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:03.030280Z","info":{"attrs":{"mtime":"2021-02-11T10:14:02.895996Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"75"}}},{"id":"1613038443.062727.ed2ca7a8","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:03.062727Z","info":{"attrs":{"mtime":"2021-02-11T10:14:02.940145Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"76"}}},{"id":"1613038443.107660.c826039e","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:03.107660Z","info":{"attrs":{"mtime":"2021-02-11T10:14:02.808877Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"73"}}},{"id":"1613038443.107823.1372ea4c","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:03.107823Z","info":{"attrs":{"mtime":"2021-02-11T10:14:02.984432Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"77"}}},{"id":"1613038443.226740.bd3c85a9","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:03.226740Z","info":{"attrs":{"mtime":"2021-02-11T10:14:03.028304Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"78"}}},{"id":"1613038443.227164.29a7afd6","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:03.227164Z","info":{"attrs":{"mtime":"2021-02-11T10:14:03.072120Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"79"}}},{"id":"1613038443.265945.71cccc46","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:03.265945Z","info":{"attrs":{"mtime":"2021-02-11T10:14:03.116480Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"80"}}},{"id":"1613038443.303597.1a1b0881","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:03.303597Z","info":{"attrs":{"mtime":"2021-02-11T10:14:03.160420Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"81"}}},{"id":"1613038443.421429.31ead625","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:03.421429Z","info":{"attrs":{"mtime":"2021-02-11T10:14:03.204199Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"82"}}},{"id":"1613038443.421625.947434fd","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:03.421625Z","info":{"attrs":{"mtime":"2021-02-11T10:14:03.296150Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"84"}}},{"id":"1613038443.470111.bba8e19d","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:03.470111Z","info":{"attrs":{"mtime":"2021-02-11T10:14:03.340288Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"85"}}},{"id":"1613038443.470287.9fcc924d","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:03.470287Z","info":{"attrs":{"mtime":"2021-02-11T10:14:03.248356Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"83"}}},{"id":"1613038443.507717.601665f9","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:03.507717Z","info":{"attrs":{"mtime":"2021-02-11T10:14:03.384655Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"86"}}},{"id":"1613038443.658308.52d35b05","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:03.658308Z","info":{"attrs":{"mtime":"2021-02-11T10:14:03.428093Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"87"}}},{"id":"1613038443.658474.273e80d3","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:03.658474Z","info":{"attrs":{"mtime":"2021-02-11T10:14:03.560487Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"90"}}},{"id":"1613038443.707954.db162408","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:03.707954Z","info":{"attrs":{"mtime":"2021-02-11T10:14:03.472055Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"88"}}},{"id":"1613038443.711967.b174e984","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:03.711967Z","info":{"attrs":{"mtime":"2021-02-11T10:14:03.516214Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"89"}}},{"id":"1613038443.751892.64ca2404","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:03.751892Z","info":{"attrs":{"mtime":"2021-02-11T10:14:03.604450Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"91"}}},{"id":"1613038443.819176.1d49d72f","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:03.819176Z","info":{"attrs":{"mtime":"2021-02-11T10:14:03.784891Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"95"}}},{"id":"1613038443.864553.67cb28be","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:03.864553Z","info":{"attrs":{"mtime":"2021-02-11T10:14:03.696038Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"93"}}},{"id":"1613038443.864832.a1fb0dd3","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:03.864832Z","info":{"attrs":{"mtime":"2021-02-11T10:14:03.740189Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"94"}}},{"id":"1613038443.911816.3410f97c","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:03.911816Z","info":{"attrs":{"mtime":"2021-02-11T10:14:03.648107Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"92"}}},{"id":"1613038443.955837.4022e4b4","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:03.955837Z","info":{"attrs":{"mtime":"2021-02-11T10:14:03.828731Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"96"}}},{"id":"1613038444.021207.9c65ad32","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:04.021207Z","info":{"attrs":{"mtime":"2021-02-11T10:14:03.964526Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"99"}}},{"id":"1613038444.024029.aeee3d0c","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:04.024029Z","info":{"attrs":{"mtime":"2021-02-11T10:14:03.920577Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"98"}}},{"id":"1613038444.024211.4d86bbe4","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:04.024211Z","info":{"attrs":{"mtime":"2021-02-11T10:14:03.876027Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"97"}}}]} 2021-02-11T10:14:05.042 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['subscription', 'pull', '--uid', 'rgw-multisite-test-user', '--subscription', 'tkscyf-85_sub', '--marker', 'tkscyf-85_sub/1613038444.024211.4d86bbe4', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:14:05.042 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'subscription', 'pull', '--uid', 'rgw-multisite-test-user', '--subscription', 'tkscyf-85_sub', '--marker', 'tkscyf-85_sub/1613038444.024211.4d86bbe4', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:14:05.042 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 subscription pull --uid rgw-multisite-test-user --subscription tkscyf-85_sub --marker tkscyf-85_sub/1613038444.024211.4d86bbe4 --rgw-zone test-zone4 --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T10:14:05.125 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:05.126 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:05.376 INFO:teuthology.orchestra.run.smithi098.stdout:{"next_marker":"","is_truncated":"false","events":[{"id":"1613038444.226676.7858b8f6","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:04.226676Z","info":{"attrs":{"mtime":"2021-02-11T10:14:04.188231Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"104"}}},{"id":"1613038444.226945.2a096e69","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:04.226945Z","info":{"attrs":{"mtime":"2021-02-11T10:14:04.144168Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"103"}}},{"id":"1613038444.270675.aa247917","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:04.270675Z","info":{"attrs":{"mtime":"2021-02-11T10:14:04.056557Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"101"}}},{"id":"1613038444.303667.9ffa5aa3","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:04.303667Z","info":{"attrs":{"mtime":"2021-02-11T10:14:04.100191Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"102"}}},{"id":"1613038444.311786.b22da454","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:04.311786Z","info":{"attrs":{"mtime":"2021-02-11T10:14:04.009121Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"100"}}},{"id":"1613038444.425761.3c1a994f","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:04.425761Z","info":{"attrs":{"mtime":"2021-02-11T10:14:04.364396Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"108"}}},{"id":"1613038444.425952.c47d708b","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:04.425952Z","info":{"attrs":{"mtime":"2021-02-11T10:14:04.232223Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"105"}}},{"id":"1613038444.468392.fc239467","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:04.468392Z","info":{"attrs":{"mtime":"2021-02-11T10:14:04.320342Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"107"}}},{"id":"1613038444.503629.fe25c5cf","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:04.503629Z","info":{"attrs":{"mtime":"2021-02-11T10:14:04.276247Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"106"}}},{"id":"1613038444.622078.0d97db55","event":"OBJECT_CREATE","timestamp":"2021-02-11T10:14:04.622078Z","info":{"attrs":{"mtime":"2021-02-11T10:14:04.408977Z"},"bucket":{"bucket_id":"dd308248-8ee8-49be-b928-07c3de7d7aa2.5484.33","name":"tkscyf-85","tenant":""},"key":{"instance":"","name":"109"}}}]} 2021-02-11T10:14:05.376 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['subscription', 'ack', '--uid', 'rgw-multisite-test-user', '--subscription', 'tkscyf-85_sub', '--event-id', '1613038444.226676.7858b8f6', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:14:05.376 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'subscription', 'ack', '--uid', 'rgw-multisite-test-user', '--subscription', 'tkscyf-85_sub', '--event-id', '1613038444.226676.7858b8f6', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:14:05.377 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 subscription ack --uid rgw-multisite-test-user --subscription tkscyf-85_sub --event-id 1613038444.226676.7858b8f6 --rgw-zone test-zone4 --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T10:14:05.460 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:05.460 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:05.733 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['subscription', 'rm', '--uid', 'rgw-multisite-test-user', '--subscription', 'tkscyf-85_sub', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:14:05.733 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'subscription', 'rm', '--uid', 'rgw-multisite-test-user', '--subscription', 'tkscyf-85_sub', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:14:05.733 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 subscription rm --uid rgw-multisite-test-user --subscription tkscyf-85_sub --rgw-zone test-zone4 --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T10:14:05.819 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:05.819 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:06.083 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['topic', 'rm', '--uid', 'rgw-multisite-test-user', '--topic', 'tkscyf-85_topic', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:14:06.083 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'topic', 'rm', '--uid', 'rgw-multisite-test-user', '--topic', 'tkscyf-85_topic', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:14:06.083 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 topic rm --uid rgw-multisite-test-user --topic tkscyf-85_topic --rgw-zone test-zone4 --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T10:14:06.166 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:06.166 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:06.437 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['topic', 'rm', '--uid', 'rgw-multisite-test-user', '--topic', 'tkscyf-85_topic', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:14:06.437 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'topic', 'rm', '--uid', 'rgw-multisite-test-user', '--topic', 'tkscyf-85_topic', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T10:14:06.437 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 topic rm --uid rgw-multisite-test-user --topic tkscyf-85_topic --rgw-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T10:14:06.478 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:06.478 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:08.054 INFO:tasks.rgw_multisite_tests:test radosgw-admin commands ... ok 2021-02-11T10:14:08.054 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:08.054 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:08.055 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:08.096 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:08.097 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:08.266 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 2021-02-11T10:14:08.301 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613038448.033697_841.1","last_update":"2021-02-11T10:14:08.033697Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:14:08.301 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T10:14:08.301 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:08.301 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:08.302 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:08.343 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:08.343 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:08.602 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038448.033697_841.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:14:08.033697Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:14:08.602 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T10:14:08.602 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T10:14:08.602 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:08.602 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:08.603 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:08.637 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:08.637 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:08.891 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038448.033697_841.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:14:08.033697Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:14:08.891 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T10:14:08.891 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:14:08.891 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:08.892 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:08.892 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:08.932 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:08.932 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:09.184 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038448.033697_841.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:14:08.033697Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:14:09.185 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:14:09.185 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:09.185 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:09.185 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:09.224 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:09.224 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:09.432 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 2021-02-11T10:14:09.474 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613038448.033697_841.1","last_update":"2021-02-11T10:14:08.033697Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:14:09.475 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:14:09.475 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:09.475 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:09.475 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:09.514 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:09.514 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:09.770 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038448.033697_841.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:14:08.033697Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:14:09.771 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:14:10.274 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-86', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:14:10.275 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-86', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:14:10.275 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-86 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:14:10.334 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:10.334 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:10.554 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:14:10.549+0000 7f9482706080 1 waiting for incremental sync to catch up: 2021-02-11T10:14:10.554 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, , , 00000000001.171.5, 00000000001.86.5, 00000000001.221.5, , 00000000001.63.5, 00000000001.143.5, 00000000001.120.5, 00000000002.107.5] 2021-02-11T10:14:10.555 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, 00000000002.119.5, , 00000000001.171.5, 00000000001.86.5, 00000000001.221.5, , 00000000001.63.5, 00000000001.143.5, 00000000001.120.5, 00000000002.107.5] 2021-02-11T10:14:15.558 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:14:15.553+0000 7f9482706080 1 bucket sync caught up with source: 2021-02-11T10:14:15.558 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, 00000000002.119.5, , 00000000001.171.5, 00000000001.86.5, 00000000001.221.5, , 00000000001.63.5, 00000000001.143.5, 00000000001.120.5, 00000000002.107.5] 2021-02-11T10:14:15.559 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, 00000000002.119.5, , 00000000001.171.5, 00000000001.86.5, 00000000001.221.5, , 00000000001.63.5, 00000000001.143.5, 00000000001.120.5, 00000000002.107.5] 2021-02-11T10:14:15.559 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:14:15.553+0000 7f9482706080 0 bucket checkpoint complete 2021-02-11T10:14:16.030 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-86', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:14:16.031 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-86', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:14:16.031 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-86 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:14:16.086 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:16.086 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:16.280 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:14:16.277+0000 7fca203bc080 1 waiting for incremental sync to catch up: 2021-02-11T10:14:16.280 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [00000000001.118.5, 00000000003.121.5, , 00000000002.173.5, 00000000001.86.5, 00000000003.225.5, , 00000000003.67.5, 00000000001.143.5, 00000000002.122.5, 00000000003.109.5] 2021-02-11T10:14:16.280 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [00000000001.118.5, 00000000003.121.5, , 00000000002.173.5, 00000000001.86.5, 00000000003.225.5, , 00000000003.67.5, 00000000002.145.5, 00000000002.122.5, 00000000003.109.5] 2021-02-11T10:14:21.284 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:14:21.281+0000 7fca203bc080 1 bucket sync caught up with source: 2021-02-11T10:14:21.284 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [00000000001.118.5, 00000000003.121.5, , 00000000002.173.5, 00000000001.86.5, 00000000003.225.5, , 00000000003.67.5, 00000000002.145.5, 00000000002.122.5, 00000000003.109.5] 2021-02-11T10:14:21.285 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [00000000001.118.5, 00000000003.121.5, , 00000000002.173.5, 00000000001.86.5, 00000000003.225.5, , 00000000003.67.5, 00000000002.145.5, 00000000002.122.5, 00000000003.109.5] 2021-02-11T10:14:21.285 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:14:21.281+0000 7fca203bc080 0 bucket checkpoint complete 2021-02-11T10:14:21.326 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "13" type: "OBJECT_CREATE" 2021-02-11T10:14:21.326 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "14" type: "OBJECT_CREATE" 2021-02-11T10:14:21.326 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "11" type: "OBJECT_CREATE" 2021-02-11T10:14:21.326 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "12" type: "OBJECT_CREATE" 2021-02-11T10:14:21.327 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "10" type: "OBJECT_CREATE" 2021-02-11T10:14:21.327 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "16" type: "OBJECT_CREATE" 2021-02-11T10:14:21.327 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "15" type: "OBJECT_CREATE" 2021-02-11T10:14:21.327 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "18" type: "OBJECT_CREATE" 2021-02-11T10:14:21.327 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "17" type: "OBJECT_CREATE" 2021-02-11T10:14:21.327 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "19" type: "OBJECT_CREATE" 2021-02-11T10:14:21.671 INFO:tasks.rgw_multisite_tests:test that events are only sent on incremental sync ... ok 2021-02-11T10:14:21.672 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:21.672 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:21.672 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:21.708 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:21.708 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:21.883 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 2021-02-11T10:14:21.913 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613038461.641893_851.1","last_update":"2021-02-11T10:14:21.641893Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:14:21.913 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T10:14:21.913 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:21.913 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:21.914 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:21.952 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:21.952 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:22.189 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038461.641893_851.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:14:21.641893Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:14:22.190 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T10:14:22.190 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T10:14:22.190 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:22.190 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:22.190 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:22.233 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:22.233 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:22.464 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038461.641893_851.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:14:21.641893Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:14:22.464 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T10:14:22.464 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:14:22.465 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:22.465 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:22.465 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:22.507 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:22.507 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:22.753 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038461.641893_851.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:14:21.641893Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:14:22.754 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:14:22.754 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:22.754 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:22.754 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:22.794 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:22.794 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:22.993 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 2021-02-11T10:14:23.032 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613038461.641893_851.1","last_update":"2021-02-11T10:14:21.641893Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:14:23.032 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:14:23.032 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:23.032 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:23.033 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:23.072 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:23.073 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:23.322 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038461.641893_851.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:14:21.641893Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:14:23.322 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:14:23.390 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:23.390 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:23.390 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:23.427 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:23.427 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:23.622 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 2021-02-11T10:14:23.657 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613038463.384997_859.1","last_update":"2021-02-11T10:14:23.384997Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:14:23.657 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:14:23.658 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:23.658 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:23.658 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:23.698 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:23.698 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:23.935 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038463.384997_859.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:14:23.384997Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:14:23.935 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:14:23.936 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-87', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:14:23.936 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-87', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:14:23.936 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-87 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:14:24.023 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:24.023 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:24.261 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:14:24.257+0000 7fabc364d080 1 bucket sync caught up with empty source 2021-02-11T10:14:24.261 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:14:24.257+0000 7fabc364d080 0 bucket checkpoint complete 2021-02-11T10:14:24.775 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-87', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:14:24.775 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-87', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:14:24.775 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-87 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:14:24.819 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:24.819 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:25.036 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:14:25.033+0000 7f2f02275080 1 waiting for incremental sync to catch up: 2021-02-11T10:14:25.036 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, 00000000002.75.5, , 00000000001.91.5, , 00000000001.119.5, , 00000000001.60.5, 00000000001.129.5, 00000000001.76.5, 00000000002.105.5] 2021-02-11T10:14:25.036 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, 00000000002.75.5, , 00000000001.91.5, 00000000001.121.5, 00000000001.119.5, , 00000000001.60.5, 00000000001.129.5, 00000000001.76.5, 00000000002.105.5] 2021-02-11T10:14:30.040 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:14:30.033+0000 7f2f02275080 1 bucket sync caught up with source: 2021-02-11T10:14:30.040 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, 00000000002.75.5, , 00000000001.91.5, 00000000001.121.5, 00000000001.119.5, , 00000000001.60.5, 00000000001.129.5, 00000000001.76.5, 00000000002.105.5] 2021-02-11T10:14:30.040 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, 00000000002.75.5, , 00000000001.91.5, 00000000001.121.5, 00000000001.119.5, , 00000000001.60.5, 00000000001.129.5, 00000000001.76.5, 00000000002.105.5] 2021-02-11T10:14:30.040 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:14:30.033+0000 7f2f02275080 0 bucket checkpoint complete 2021-02-11T10:14:30.081 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "1" type: "OBJECT_CREATE" 2021-02-11T10:14:30.081 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "0" type: "OBJECT_CREATE" 2021-02-11T10:14:30.081 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "2" type: "OBJECT_CREATE" 2021-02-11T10:14:30.081 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "5" type: "OBJECT_CREATE" 2021-02-11T10:14:30.081 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "6" type: "OBJECT_CREATE" 2021-02-11T10:14:30.081 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "9" type: "OBJECT_CREATE" 2021-02-11T10:14:30.081 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "8" type: "OBJECT_CREATE" 2021-02-11T10:14:30.082 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "4" type: "OBJECT_CREATE" 2021-02-11T10:14:30.082 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "3" type: "OBJECT_CREATE" 2021-02-11T10:14:30.082 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "7" type: "OBJECT_CREATE" 2021-02-11T10:14:30.093 DEBUG:tasks.rgw_multi.tests_ps:superfluous events are found 2021-02-11T10:14:30.107 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "1" type: "OBJECT_CREATE" 2021-02-11T10:14:30.107 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "0" type: "OBJECT_CREATE" 2021-02-11T10:14:30.107 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "2" type: "OBJECT_CREATE" 2021-02-11T10:14:30.107 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "5" type: "OBJECT_CREATE" 2021-02-11T10:14:30.107 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "6" type: "OBJECT_CREATE" 2021-02-11T10:14:30.107 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "9" type: "OBJECT_CREATE" 2021-02-11T10:14:30.107 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "8" type: "OBJECT_CREATE" 2021-02-11T10:14:30.107 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "4" type: "OBJECT_CREATE" 2021-02-11T10:14:30.107 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "3" type: "OBJECT_CREATE" 2021-02-11T10:14:30.107 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "7" type: "OBJECT_CREATE" 2021-02-11T10:14:30.107 DEBUG:tasks.rgw_multi.tests_ps:superfluous events are found 2021-02-11T10:14:30.196 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-87', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:14:30.197 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-87', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:14:30.197 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-87 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:14:30.257 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:30.258 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:30.483 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:14:30.477+0000 7f74cabe6080 1 bucket sync caught up with source: 2021-02-11T10:14:30.484 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, 00000000004.79.5, , 00000000002.93.5, 00000000002.125.5, 00000000002.123.5, , 00000000002.62.5, 00000000002.131.5, 00000000002.78.5, 00000000004.109.5] 2021-02-11T10:14:30.484 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, 00000000004.79.5, , 00000000002.93.5, 00000000002.125.5, 00000000002.123.5, , 00000000002.62.5, 00000000002.131.5, 00000000002.78.5, 00000000004.109.5] 2021-02-11T10:14:30.484 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:14:30.477+0000 7f74cabe6080 0 bucket checkpoint complete 2021-02-11T10:14:30.512 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_DELETE) synced 2021-02-11T10:14:30.520 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "1" type: "OBJECT_CREATE" 2021-02-11T10:14:30.520 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "0" type: "OBJECT_CREATE" 2021-02-11T10:14:30.520 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "2" type: "OBJECT_CREATE" 2021-02-11T10:14:30.520 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "5" type: "OBJECT_CREATE" 2021-02-11T10:14:30.520 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "6" type: "OBJECT_CREATE" 2021-02-11T10:14:30.520 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "9" type: "OBJECT_CREATE" 2021-02-11T10:14:30.520 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "8" type: "OBJECT_CREATE" 2021-02-11T10:14:30.520 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "4" type: "OBJECT_CREATE" 2021-02-11T10:14:30.520 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "3" type: "OBJECT_CREATE" 2021-02-11T10:14:30.520 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "7" type: "OBJECT_CREATE" 2021-02-11T10:14:30.520 DEBUG:tasks.rgw_multi.tests_ps:superfluous events are found 2021-02-11T10:14:30.528 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_DELETE): objname: "3" type: "OBJECT_DELETE" 2021-02-11T10:14:30.528 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_DELETE): objname: "4" type: "OBJECT_DELETE" 2021-02-11T10:14:30.528 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_DELETE): objname: "8" type: "OBJECT_DELETE" 2021-02-11T10:14:30.528 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_DELETE): objname: "5" type: "OBJECT_DELETE" 2021-02-11T10:14:30.529 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_DELETE): objname: "9" type: "OBJECT_DELETE" 2021-02-11T10:14:30.529 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_DELETE): objname: "6" type: "OBJECT_DELETE" 2021-02-11T10:14:30.529 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_DELETE): objname: "2" type: "OBJECT_DELETE" 2021-02-11T10:14:30.529 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_DELETE): objname: "1" type: "OBJECT_DELETE" 2021-02-11T10:14:30.529 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_DELETE): objname: "0" type: "OBJECT_DELETE" 2021-02-11T10:14:30.529 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_DELETE): objname: "7" type: "OBJECT_DELETE" 2021-02-11T10:14:30.529 DEBUG:tasks.rgw_multi.tests_ps:superfluous events are found 2021-02-11T10:14:30.536 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "1" type: "OBJECT_CREATE" 2021-02-11T10:14:30.536 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "0" type: "OBJECT_CREATE" 2021-02-11T10:14:30.536 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "2" type: "OBJECT_CREATE" 2021-02-11T10:14:30.536 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "5" type: "OBJECT_CREATE" 2021-02-11T10:14:30.536 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "6" type: "OBJECT_CREATE" 2021-02-11T10:14:30.536 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "9" type: "OBJECT_CREATE" 2021-02-11T10:14:30.536 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "8" type: "OBJECT_CREATE" 2021-02-11T10:14:30.536 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "4" type: "OBJECT_CREATE" 2021-02-11T10:14:30.536 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "3" type: "OBJECT_CREATE" 2021-02-11T10:14:30.537 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "7" type: "OBJECT_CREATE" 2021-02-11T10:14:30.537 DEBUG:tasks.rgw_multi.tests_ps:superfluous events are found 2021-02-11T10:14:30.718 INFO:tasks.rgw_multisite_tests:test subscriptions for different events ... ok 2021-02-11T10:14:30.718 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:30.719 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:30.719 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:30.756 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:30.756 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:30.951 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 2021-02-11T10:14:30.990 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613038470.704193_867.1","last_update":"2021-02-11T10:14:30.704193Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:14:30.990 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T10:14:30.991 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:30.991 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:30.991 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:31.031 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:31.031 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:31.289 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038470.704193_867.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:14:30.704193Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:14:31.290 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T10:14:31.290 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T10:14:31.290 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:31.290 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:31.290 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:31.340 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:31.340 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:31.559 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038470.704193_867.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:14:30.704193Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:14:31.560 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T10:14:31.560 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:14:31.560 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:31.560 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:31.560 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:31.601 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:31.601 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:31.871 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038470.704193_867.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:14:30.704193Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:14:31.872 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:14:31.872 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:31.872 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:31.872 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:31.911 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:31.911 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:32.139 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 2021-02-11T10:14:32.180 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613038470.704193_867.1","last_update":"2021-02-11T10:14:30.704193Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:14:32.180 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:14:32.180 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:32.180 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:32.181 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:32.221 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:32.221 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:32.473 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038470.704193_867.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:14:30.704193Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:14:32.473 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:14:32.530 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:32.530 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:32.531 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:32.564 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:32.564 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:32.765 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 2021-02-11T10:14:32.805 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613038472.525881_873.1","last_update":"2021-02-11T10:14:32.525881Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:14:32.805 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:14:32.805 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:32.805 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:32.806 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:32.849 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:32.849 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:33.108 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038472.525881_873.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:14:32.525881Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:14:33.109 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:14:37.543 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-88', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:14:37.543 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-88', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:14:37.543 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-88 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:14:37.586 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:37.586 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:37.836 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:14:37.833+0000 7eff0bf01080 1 bucket sync caught up with source: 2021-02-11T10:14:37.836 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [00000000010.113.5, 00000000010.141.5, 00000000009.106.5, 00000000010.116.5, 00000000007.94.5, 00000000007.155.5, 00000000008.79.5, 00000000010.252.5, 00000000010.118.5, 00000000010.114.5, 00000000009.133.5] 2021-02-11T10:14:37.836 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [00000000010.113.5, 00000000010.141.5, 00000000009.106.5, 00000000010.116.5, 00000000007.94.5, 00000000007.155.5, 00000000008.79.5, 00000000010.252.5, 00000000010.118.5, 00000000010.114.5, 00000000009.133.5] 2021-02-11T10:14:37.836 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:14:37.833+0000 7eff0bf01080 0 bucket checkpoint complete 2021-02-11T10:14:37.884 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "2" type: "OBJECT_CREATE" 2021-02-11T10:14:37.884 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "0" type: "OBJECT_CREATE" 2021-02-11T10:14:37.884 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "1" type: "OBJECT_CREATE" 2021-02-11T10:14:37.884 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "5" type: "OBJECT_CREATE" 2021-02-11T10:14:37.884 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "6" type: "OBJECT_CREATE" 2021-02-11T10:14:37.884 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "9" type: "OBJECT_CREATE" 2021-02-11T10:14:37.884 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "11" type: "OBJECT_CREATE" 2021-02-11T10:14:37.884 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "3" type: "OBJECT_CREATE" 2021-02-11T10:14:37.884 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "10" type: "OBJECT_CREATE" 2021-02-11T10:14:37.884 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "4" type: "OBJECT_CREATE" 2021-02-11T10:14:37.884 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "7" type: "OBJECT_CREATE" 2021-02-11T10:14:37.885 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "12" type: "OBJECT_CREATE" 2021-02-11T10:14:37.885 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "8" type: "OBJECT_CREATE" 2021-02-11T10:14:37.885 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "15" type: "OBJECT_CREATE" 2021-02-11T10:14:37.885 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "13" type: "OBJECT_CREATE" 2021-02-11T10:14:37.894 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "14" type: "OBJECT_CREATE" 2021-02-11T10:14:37.894 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "16" type: "OBJECT_CREATE" 2021-02-11T10:14:37.894 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "20" type: "OBJECT_CREATE" 2021-02-11T10:14:37.894 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "18" type: "OBJECT_CREATE" 2021-02-11T10:14:37.894 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "17" type: "OBJECT_CREATE" 2021-02-11T10:14:37.894 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "19" type: "OBJECT_CREATE" 2021-02-11T10:14:37.895 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "21" type: "OBJECT_CREATE" 2021-02-11T10:14:37.895 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "23" type: "OBJECT_CREATE" 2021-02-11T10:14:37.895 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "22" type: "OBJECT_CREATE" 2021-02-11T10:14:37.895 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "24" type: "OBJECT_CREATE" 2021-02-11T10:14:37.895 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "28" type: "OBJECT_CREATE" 2021-02-11T10:14:37.895 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "29" type: "OBJECT_CREATE" 2021-02-11T10:14:37.895 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "25" type: "OBJECT_CREATE" 2021-02-11T10:14:37.895 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "26" type: "OBJECT_CREATE" 2021-02-11T10:14:37.895 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "27" type: "OBJECT_CREATE" 2021-02-11T10:14:37.902 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "33" type: "OBJECT_CREATE" 2021-02-11T10:14:37.902 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "32" type: "OBJECT_CREATE" 2021-02-11T10:14:37.903 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "31" type: "OBJECT_CREATE" 2021-02-11T10:14:37.903 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "30" type: "OBJECT_CREATE" 2021-02-11T10:14:37.903 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "35" type: "OBJECT_CREATE" 2021-02-11T10:14:37.903 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "37" type: "OBJECT_CREATE" 2021-02-11T10:14:37.903 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "34" type: "OBJECT_CREATE" 2021-02-11T10:14:37.903 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "36" type: "OBJECT_CREATE" 2021-02-11T10:14:37.903 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "38" type: "OBJECT_CREATE" 2021-02-11T10:14:37.903 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "41" type: "OBJECT_CREATE" 2021-02-11T10:14:37.903 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "39" type: "OBJECT_CREATE" 2021-02-11T10:14:37.903 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "42" type: "OBJECT_CREATE" 2021-02-11T10:14:37.903 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "40" type: "OBJECT_CREATE" 2021-02-11T10:14:37.903 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "44" type: "OBJECT_CREATE" 2021-02-11T10:14:37.903 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "47" type: "OBJECT_CREATE" 2021-02-11T10:14:37.912 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "43" type: "OBJECT_CREATE" 2021-02-11T10:14:37.912 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "45" type: "OBJECT_CREATE" 2021-02-11T10:14:37.912 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "46" type: "OBJECT_CREATE" 2021-02-11T10:14:37.912 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "48" type: "OBJECT_CREATE" 2021-02-11T10:14:37.912 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "50" type: "OBJECT_CREATE" 2021-02-11T10:14:37.912 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "49" type: "OBJECT_CREATE" 2021-02-11T10:14:37.912 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "51" type: "OBJECT_CREATE" 2021-02-11T10:14:37.912 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "52" type: "OBJECT_CREATE" 2021-02-11T10:14:37.912 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "54" type: "OBJECT_CREATE" 2021-02-11T10:14:37.912 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "53" type: "OBJECT_CREATE" 2021-02-11T10:14:37.912 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "55" type: "OBJECT_CREATE" 2021-02-11T10:14:37.912 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "56" type: "OBJECT_CREATE" 2021-02-11T10:14:37.912 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "59" type: "OBJECT_CREATE" 2021-02-11T10:14:37.913 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "58" type: "OBJECT_CREATE" 2021-02-11T10:14:37.913 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "60" type: "OBJECT_CREATE" 2021-02-11T10:14:37.923 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "57" type: "OBJECT_CREATE" 2021-02-11T10:14:37.923 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "61" type: "OBJECT_CREATE" 2021-02-11T10:14:37.923 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "63" type: "OBJECT_CREATE" 2021-02-11T10:14:37.923 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "64" type: "OBJECT_CREATE" 2021-02-11T10:14:37.923 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "65" type: "OBJECT_CREATE" 2021-02-11T10:14:37.923 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "62" type: "OBJECT_CREATE" 2021-02-11T10:14:37.924 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "66" type: "OBJECT_CREATE" 2021-02-11T10:14:37.924 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "67" type: "OBJECT_CREATE" 2021-02-11T10:14:37.924 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "70" type: "OBJECT_CREATE" 2021-02-11T10:14:37.924 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "71" type: "OBJECT_CREATE" 2021-02-11T10:14:37.924 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "68" type: "OBJECT_CREATE" 2021-02-11T10:14:37.924 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "69" type: "OBJECT_CREATE" 2021-02-11T10:14:37.924 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "74" type: "OBJECT_CREATE" 2021-02-11T10:14:37.924 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "72" type: "OBJECT_CREATE" 2021-02-11T10:14:37.924 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "73" type: "OBJECT_CREATE" 2021-02-11T10:14:37.930 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "77" type: "OBJECT_CREATE" 2021-02-11T10:14:37.930 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "75" type: "OBJECT_CREATE" 2021-02-11T10:14:37.931 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "79" type: "OBJECT_CREATE" 2021-02-11T10:14:37.931 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "76" type: "OBJECT_CREATE" 2021-02-11T10:14:37.931 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "78" type: "OBJECT_CREATE" 2021-02-11T10:14:37.931 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "82" type: "OBJECT_CREATE" 2021-02-11T10:14:37.931 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "83" type: "OBJECT_CREATE" 2021-02-11T10:14:37.931 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "81" type: "OBJECT_CREATE" 2021-02-11T10:14:37.931 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "80" type: "OBJECT_CREATE" 2021-02-11T10:14:37.931 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "85" type: "OBJECT_CREATE" 2021-02-11T10:14:37.931 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "84" type: "OBJECT_CREATE" 2021-02-11T10:14:37.931 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "87" type: "OBJECT_CREATE" 2021-02-11T10:14:37.931 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "86" type: "OBJECT_CREATE" 2021-02-11T10:14:37.931 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "90" type: "OBJECT_CREATE" 2021-02-11T10:14:37.931 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "89" type: "OBJECT_CREATE" 2021-02-11T10:14:37.938 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "92" type: "OBJECT_CREATE" 2021-02-11T10:14:37.938 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "91" type: "OBJECT_CREATE" 2021-02-11T10:14:37.938 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "88" type: "OBJECT_CREATE" 2021-02-11T10:14:37.938 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "93" type: "OBJECT_CREATE" 2021-02-11T10:14:37.939 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "96" type: "OBJECT_CREATE" 2021-02-11T10:14:37.939 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "97" type: "OBJECT_CREATE" 2021-02-11T10:14:37.939 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "95" type: "OBJECT_CREATE" 2021-02-11T10:14:37.939 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "94" type: "OBJECT_CREATE" 2021-02-11T10:14:37.939 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "98" type: "OBJECT_CREATE" 2021-02-11T10:14:37.939 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "99" type: "OBJECT_CREATE" 2021-02-11T10:14:37.959 DEBUG:tasks.rgw_multi.tests_ps:superfluous events are found 2021-02-11T10:14:39.186 INFO:tasks.rgw_multisite_tests:test incremental fetching of events from a subscription ... ok 2021-02-11T10:14:43.333 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:43.334 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:43.334 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:43.367 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:43.367 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:43.572 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 2021-02-11T10:14:43.618 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613038479.173882_900.1","last_update":"2021-02-11T10:14:39.173882Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:14:43.619 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T10:14:43.619 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:43.619 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:43.619 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:43.661 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:43.661 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:43.933 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038479.173882_900.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:14:39.173882Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:14:43.934 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T10:14:43.934 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T10:14:43.934 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:43.934 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:43.934 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:43.975 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:43.975 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:44.225 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038479.173882_900.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:14:39.173882Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:14:44.225 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T10:14:44.225 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:14:44.226 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:44.226 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:44.226 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:44.269 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:44.269 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:44.495 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038479.173882_900.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:14:39.173882Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:14:44.496 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:14:44.496 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:44.496 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:44.496 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:44.535 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:44.535 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:44.753 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 2021-02-11T10:14:44.810 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613038479.173882_900.1","last_update":"2021-02-11T10:14:39.173882Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:14:44.810 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:14:44.810 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:44.811 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:44.811 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:44.850 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:44.850 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:45.103 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038479.173882_900.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:14:39.173882Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:14:45.103 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:14:45.162 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:45.162 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:45.163 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:45.196 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:45.196 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:45.401 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 2021-02-11T10:14:45.442 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613038485.157744_908.1","last_update":"2021-02-11T10:14:45.157744Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:14:45.443 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:14:45.443 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:45.443 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:45.443 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:45.483 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:45.484 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:45.738 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038485.157744_908.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:14:45.157744Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:14:45.738 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:14:46.175 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-89', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:14:46.175 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-89', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:14:46.175 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-89 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:14:46.216 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:46.216 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:46.457 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:14:46.453+0000 7f46f1e26080 1 waiting for incremental sync to catch up: 2021-02-11T10:14:46.457 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, 00000000002.127.5, , 00000000001.163.5, 00000000001.165.5, 00000000001.164.5, , , 00000000001.112.5, 00000000001.88.5, 00000000002.164.5] 2021-02-11T10:14:46.457 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, 00000000002.127.5, , 00000000001.163.5, 00000000001.165.5, 00000000001.164.5, , 00000000001.180.5, 00000000001.112.5, 00000000001.88.5, 00000000002.164.5] 2021-02-11T10:14:51.462 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:14:51.457+0000 7f46f1e26080 1 bucket sync caught up with source: 2021-02-11T10:14:51.462 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, 00000000002.127.5, , 00000000001.163.5, 00000000001.165.5, 00000000001.164.5, , 00000000001.180.5, 00000000001.112.5, 00000000001.88.5, 00000000002.164.5] 2021-02-11T10:14:51.462 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, 00000000002.127.5, , 00000000001.163.5, 00000000001.165.5, 00000000001.164.5, , 00000000001.180.5, 00000000001.112.5, 00000000001.88.5, 00000000002.164.5] 2021-02-11T10:14:51.462 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:14:51.457+0000 7f46f1e26080 0 bucket checkpoint complete 2021-02-11T10:14:51.508 DEBUG:tasks.rgw_multi.tests_ps:Event (before ack) id: "1613038485.946806.e6bb062c" 2021-02-11T10:14:51.508 DEBUG:tasks.rgw_multi.tests_ps:Event (before ack) id: "1613038485.947488.2142da8f" 2021-02-11T10:14:51.508 DEBUG:tasks.rgw_multi.tests_ps:Event (before ack) id: "1613038486.092533.a565e201" 2021-02-11T10:14:51.508 DEBUG:tasks.rgw_multi.tests_ps:Event (before ack) id: "1613038486.168457.dedfc3ea" 2021-02-11T10:14:51.509 DEBUG:tasks.rgw_multi.tests_ps:Event (before ack) id: "1613038486.231930.53dea0bd" 2021-02-11T10:14:51.509 DEBUG:tasks.rgw_multi.tests_ps:Event (before ack) id: "1613038486.287515.94701a32" 2021-02-11T10:14:51.509 DEBUG:tasks.rgw_multi.tests_ps:Event (before ack) id: "1613038486.287701.6f333478" 2021-02-11T10:14:51.509 DEBUG:tasks.rgw_multi.tests_ps:Event (before ack) id: "1613038486.330022.4925050d" 2021-02-11T10:14:51.509 DEBUG:tasks.rgw_multi.tests_ps:Event (before ack) id: "1613038486.379730.ab6c0371" 2021-02-11T10:14:51.509 DEBUG:tasks.rgw_multi.tests_ps:Event (before ack) id: "1613038486.928525.47250adf" 2021-02-11T10:14:51.523 DEBUG:tasks.rgw_multi.tests_ps:superfluous events are found 2021-02-11T10:14:51.573 DEBUG:tasks.rgw_multi.tests_ps:Event (after ack) id: "1613038486.287515.94701a32" 2021-02-11T10:14:51.574 DEBUG:tasks.rgw_multi.tests_ps:Event (after ack) id: "1613038486.287701.6f333478" 2021-02-11T10:14:51.574 DEBUG:tasks.rgw_multi.tests_ps:Event (after ack) id: "1613038486.330022.4925050d" 2021-02-11T10:14:51.574 DEBUG:tasks.rgw_multi.tests_ps:Event (after ack) id: "1613038486.379730.ab6c0371" 2021-02-11T10:14:51.574 DEBUG:tasks.rgw_multi.tests_ps:Event (after ack) id: "1613038486.928525.47250adf" 2021-02-11T10:14:51.772 INFO:tasks.rgw_multisite_tests:test acking of some events in a subscription ... ok 2021-02-11T10:14:51.772 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:51.773 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:51.773 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:51.810 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:51.811 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:52.020 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 2021-02-11T10:14:52.068 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613038491.759800_911.1","last_update":"2021-02-11T10:14:51.759800Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:14:52.068 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T10:14:52.068 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:52.068 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:52.068 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:52.105 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:52.105 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:52.361 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038491.759800_911.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:14:51.759800Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:14:52.362 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T10:14:52.362 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T10:14:52.362 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:52.362 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:52.363 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:52.403 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:52.403 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:52.656 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038491.759800_911.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:14:51.759800Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:14:52.656 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T10:14:52.656 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:14:52.656 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:52.656 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:52.656 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:52.697 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:52.697 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:52.963 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038491.759800_911.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:14:51.759800Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:14:52.964 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:14:52.964 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:52.964 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:52.964 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:53.003 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:53.003 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:53.214 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 2021-02-11T10:14:53.256 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613038491.759800_911.1","last_update":"2021-02-11T10:14:51.759800Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:14:53.256 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:14:53.256 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:53.256 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:53.257 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:53.297 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:53.297 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:53.523 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038491.759800_911.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:14:51.759800Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:14:53.523 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:14:53.585 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:53.585 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:53.585 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:53.620 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:53.620 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:53.806 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 2021-02-11T10:14:53.843 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613038493.579276_918.1","last_update":"2021-02-11T10:14:53.579276Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:14:53.843 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:14:53.844 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:53.844 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:53.844 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:53.884 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:53.884 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:54.116 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038493.579276_918.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:14:53.579276Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:14:54.116 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:14:54.431 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-90', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:14:54.431 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-90', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:14:54.431 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-90 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:14:54.479 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:54.479 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:54.685 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:14:54.681+0000 7fd85dbb3080 1 bucket sync caught up with source: 2021-02-11T10:14:54.685 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [00000000001.123.5, , , , , , 00000000001.89.5, , , 00000000006.134.5, ] 2021-02-11T10:14:54.685 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [00000000001.123.5, , , , , , 00000000001.89.5, , , 00000000006.134.5, ] 2021-02-11T10:14:54.685 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:14:54.681+0000 7fd85dbb3080 0 bucket checkpoint complete 2021-02-11T10:14:54.729 DEBUG:tasks.rgw_multi.tests_ps:Event key: "put" type: "OBJECT_CREATE" 2021-02-11T10:14:54.729 DEBUG:tasks.rgw_multi.tests_ps:Event key: "copy" type: "OBJECT_CREATE" 2021-02-11T10:14:54.729 DEBUG:tasks.rgw_multi.tests_ps:Event key: "multipart" type: "OBJECT_CREATE" 2021-02-11T10:14:54.883 INFO:tasks.rgw_multisite_tests:test object creation notifications in using put/copy/post ... ok 2021-02-11T10:14:54.884 INFO:tasks.rgw_multisite_tests:test object creation s3 notifications in using put/copy/post on master ... ok 2021-02-11T10:14:54.884 INFO:tasks.rgw_multisite_tests:test multipart object upload on master ... ok 2021-02-11T10:14:54.884 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:54.884 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:54.884 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:54.920 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:54.920 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:55.115 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 2021-02-11T10:14:55.148 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613038494.873664_921.1","last_update":"2021-02-11T10:14:54.873664Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:14:55.149 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T10:14:55.149 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:55.149 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:55.150 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:55.187 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:55.188 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:55.441 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038494.873664_921.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:14:54.873664Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:14:55.441 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T10:14:55.441 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T10:14:55.441 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:55.441 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:55.442 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:55.481 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:55.481 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:55.727 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038494.873664_921.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:14:54.873664Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:14:55.727 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T10:14:55.727 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:14:55.727 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:55.728 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:55.728 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:55.769 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:55.769 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:56.020 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038494.873664_921.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:14:54.873664Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:14:56.021 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:14:56.021 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:56.021 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:56.021 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:56.061 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:56.061 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:56.263 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 2021-02-11T10:14:56.296 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613038494.873664_921.1","last_update":"2021-02-11T10:14:54.873664Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:14:56.296 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:14:56.296 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:56.297 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:56.297 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:56.337 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:56.338 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:56.596 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038494.873664_921.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:14:54.873664Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:14:56.597 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:14:56.661 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:56.661 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:56.661 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:56.699 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:56.699 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:56.890 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 2021-02-11T10:14:56.929 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613038496.656571_929.1","last_update":"2021-02-11T10:14:56.656571Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:14:56.929 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:14:56.929 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:56.930 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:14:56.930 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:14:56.971 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:56.971 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:57.215 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038496.656571_929.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:14:56.656571Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:14:57.216 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:14:57.351 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-91', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:14:57.352 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-91', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:14:57.352 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-91 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:14:57.404 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:57.405 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:57.616 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:14:57.612+0000 7f54946be080 1 bucket sync caught up with source: 2021-02-11T10:14:57.617 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, , , , , , , , , , 00000000005.109.11] 2021-02-11T10:14:57.617 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, , , , , , , , , , 00000000005.109.11] 2021-02-11T10:14:57.617 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:14:57.612+0000 7f54946be080 0 bucket checkpoint complete 2021-02-11T10:14:57.722 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-91', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:14:57.722 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-91', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:14:57.723 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-91 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:14:57.774 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:57.774 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:58.006 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:14:58.000+0000 7f2f74587080 1 bucket sync caught up with source: 2021-02-11T10:14:58.007 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, , , , , , , , , , 00000000011.122.10] 2021-02-11T10:14:58.007 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, , , , , , , , , , 00000000011.122.10] 2021-02-11T10:14:58.007 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:14:58.000+0000 7f2f74587080 0 bucket checkpoint complete 2021-02-11T10:14:58.049 DEBUG:tasks.rgw_multi.tests_ps:Event key: "foo" type: "OBJECT_DELETE" 2021-02-11T10:14:58.049 DEBUG:tasks.rgw_multi.tests_ps:Event key: "foo" type: "OBJECT_DELETE" 2021-02-11T10:14:58.049 DEBUG:tasks.rgw_multi.tests_ps:Event key: "foo" type: "OBJECT_DELETE" 2021-02-11T10:14:58.057 DEBUG:tasks.rgw_multi.tests_ps:Event key: "foo" type: "DELETE_MARKER_CREATE" 2021-02-11T10:14:58.058 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-91', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:14:58.058 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-91', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:14:58.058 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-91 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:14:58.102 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:58.102 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:58.348 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:14:58.340+0000 7f14b736f080 1 bucket sync caught up with source: 2021-02-11T10:14:58.348 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, , , , , , , , , , 00000000011.122.10] 2021-02-11T10:14:58.348 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, , , , , , , , , , 00000000011.122.10] 2021-02-11T10:14:58.348 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:14:58.340+0000 7f14b736f080 0 bucket checkpoint complete 2021-02-11T10:14:58.379 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-91', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:14:58.379 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-91', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:14:58.379 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-91 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:14:58.420 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:58.420 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:58.640 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:14:58.637+0000 7fa032ef8080 1 bucket sync caught up with source: 2021-02-11T10:14:58.640 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , , , 00000000011.122.10] 2021-02-11T10:14:58.640 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , , , 00000000011.122.10] 2021-02-11T10:14:58.640 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:14:58.637+0000 7fa032ef8080 0 bucket checkpoint complete 2021-02-11T10:14:58.666 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-91', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:14:58.666 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-91', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:14:58.666 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-91 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:14:58.713 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:58.714 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:58.959 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:14:58.952+0000 7f9221fc9080 1 bucket sync caught up with source: 2021-02-11T10:14:58.959 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, , , , , , , , , , 00000000011.122.10] 2021-02-11T10:14:58.959 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, , , , , , , , , , 00000000011.122.10] 2021-02-11T10:14:58.959 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:14:58.952+0000 7f9221fc9080 0 bucket checkpoint complete 2021-02-11T10:14:58.994 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-91', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:14:58.994 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-91', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:14:58.995 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-91 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:14:59.042 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:59.042 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:59.275 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:14:59.273+0000 7fee4f954080 1 bucket sync caught up with source: 2021-02-11T10:14:59.275 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , , , 00000000008.95.5] 2021-02-11T10:14:59.275 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , , , 00000000008.95.5] 2021-02-11T10:14:59.275 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:14:59.273+0000 7fee4f954080 0 bucket checkpoint complete 2021-02-11T10:14:59.309 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-91', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:14:59.309 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-91', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:14:59.310 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-91 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:14:59.354 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:59.354 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:59.589 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:14:59.585+0000 7efe9ec37080 1 bucket sync caught up with source: 2021-02-11T10:14:59.589 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , , , 00000000008.95.5] 2021-02-11T10:14:59.589 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , , , 00000000008.95.5] 2021-02-11T10:14:59.589 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:14:59.585+0000 7efe9ec37080 0 bucket checkpoint complete 2021-02-11T10:14:59.623 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-91', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:14:59.623 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-91', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:14:59.623 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket tkscyf-91 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T10:14:59.713 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:14:59.713 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:14:59.947 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:14:59.945+0000 7f37c4c42080 1 bucket sync caught up with source: 2021-02-11T10:14:59.947 INFO:teuthology.orchestra.run.smithi094.stderr: local status: [, , , , , , , , , , 00000000008.98.5] 2021-02-11T10:14:59.947 INFO:teuthology.orchestra.run.smithi094.stderr: remote markers: [, , , , , , , , , , 00000000008.98.5] 2021-02-11T10:14:59.947 INFO:teuthology.orchestra.run.smithi094.stderr:2021-02-11T10:14:59.945+0000 7f37c4c42080 0 bucket checkpoint complete 2021-02-11T10:14:59.977 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-91', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:14:59.977 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-91', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:14:59.978 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-91 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:15:00.024 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:00.024 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:00.275 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:15:00.268+0000 7fcdf9202080 1 bucket sync caught up with source: 2021-02-11T10:15:00.276 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, , , , , , , , , , 00000000008.98.5] 2021-02-11T10:15:00.276 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, , , , , , , , , , 00000000008.98.5] 2021-02-11T10:15:00.276 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:15:00.268+0000 7fcdf9202080 0 bucket checkpoint complete 2021-02-11T10:15:00.312 INFO:tasks.rgw_multi.zone_rados:comparing bucket=tkscyf-91 zones={test-zone2, test-zone1} 2021-02-11T10:15:00.329 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T10:15:00.378 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T10:15:00.385 DEBUG:tasks.rgw_multi.zone_rados:o=foo 2021-02-11T10:15:00.439 CRITICAL:tasks.rgw_multi.zone_rados:key=foo is missing from zone=test-zone2 2021-02-11T10:15:00.440 DEBUG:tasks.rgw_multi.tests_ps:zonegroup_bucket_checkpoint failed, cannot delete bucket 2021-02-11T10:15:00.498 INFO:tasks.rgw_multisite_tests:test notification of deletion markers ... ok 2021-02-11T10:15:00.499 INFO:tasks.rgw_multisite_tests:test s3 notification of metadata on master ... ok 2021-02-11T10:15:00.499 INFO:tasks.rgw_multisite_tests:test s3 notification of tags on master ... ok 2021-02-11T10:15:00.499 INFO:tasks.rgw_multisite_tests:test s3 notification of object versions ... ok 2021-02-11T10:15:00.499 INFO:tasks.rgw_multisite_tests:test s3 notification of deletion markers on master ... ok 2021-02-11T10:15:00.499 INFO:tasks.rgw_multisite_tests:test pushing to http endpoint ... ok 2021-02-11T10:15:00.500 INFO:tasks.rgw_multisite_tests:test pushing to http endpoint s3 record format ... ok 2021-02-11T10:15:00.500 INFO:tasks.rgw_multisite_tests:test pushing to amqp endpoint ... ok 2021-02-11T10:15:00.500 INFO:tasks.rgw_multisite_tests:test pushing to amqp endpoint s3 record format ... ok 2021-02-11T10:15:00.500 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:00.500 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:00.500 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:00.540 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:00.540 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:00.755 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 2021-02-11T10:15:00.794 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613038496.656571_929.1","last_update":"2021-02-11T10:14:56.656571Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:15:00.794 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T10:15:00.794 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:00.794 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:00.794 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:00.834 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:00.834 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:01.100 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038496.656571_929.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:14:56.656571Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:15:01.100 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T10:15:01.100 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T10:15:01.101 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:01.101 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:01.101 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:01.143 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:01.143 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:01.367 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038496.656571_929.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:14:56.656571Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:15:01.368 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T10:15:01.368 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:15:01.368 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:01.368 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:01.368 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:01.408 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:01.408 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:01.650 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038496.656571_929.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:14:56.656571Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:15:01.651 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:15:01.651 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:01.651 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:01.651 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:01.690 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:01.690 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:01.890 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 2021-02-11T10:15:01.924 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613038496.656571_929.1","last_update":"2021-02-11T10:14:56.656571Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:15:01.924 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:15:01.925 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:01.925 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:01.925 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:01.965 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:01.965 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:02.223 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038496.656571_929.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:14:56.656571Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:15:02.223 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:15:02.281 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:02.281 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:02.281 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:02.318 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:02.318 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:02.519 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 2021-02-11T10:15:02.547 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613038502.276108_938.1","last_update":"2021-02-11T10:15:02.276108Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:15:02.547 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:15:02.548 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:02.548 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:02.548 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:02.588 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:02.588 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:02.823 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038502.276108_938.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:15:02.276108Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:15:02.824 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:15:03.291 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-92', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:15:03.291 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-92', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:15:03.291 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-92 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:15:03.347 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:03.347 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:03.582 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:15:03.576+0000 7f70a7e03080 1 bucket sync caught up with source: 2021-02-11T10:15:03.582 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, 00000000002.98.5, , 00000000001.95.5, 00000000001.140.5, 00000000001.131.5, , 00000000001.138.5, 00000000001.153.5, 00000000001.97.5, 00000000003.112.3] 2021-02-11T10:15:03.582 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, 00000000002.98.5, , 00000000001.95.5, 00000000001.140.5, 00000000001.131.5, , 00000000001.138.5, 00000000001.153.5, 00000000001.97.5, 00000000003.112.3] 2021-02-11T10:15:03.582 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:15:03.576+0000 7f70a7e03080 0 bucket checkpoint complete 2021-02-11T10:15:03.703 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-92', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:15:03.704 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'tkscyf-92', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T10:15:03.704 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket tkscyf-92 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T10:15:03.755 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:03.755 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:04.012 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:15:04.008+0000 7f2ff6e72080 1 bucket sync caught up with source: 2021-02-11T10:15:04.012 INFO:teuthology.orchestra.run.smithi098.stderr: local status: [, 00000000004.102.5, , 00000000002.99.5, 00000000002.144.5, 00000000002.133.5, , 00000000002.142.5, 00000000002.155.5, 00000000002.101.5, 00000000005.116.5] 2021-02-11T10:15:04.012 INFO:teuthology.orchestra.run.smithi098.stderr: remote markers: [, 00000000004.102.5, , 00000000002.99.5, 00000000002.144.5, 00000000002.133.5, , 00000000002.142.5, 00000000002.155.5, 00000000002.101.5, 00000000005.116.5] 2021-02-11T10:15:04.012 INFO:teuthology.orchestra.run.smithi098.stderr:2021-02-11T10:15:04.008+0000 7f2ff6e72080 0 bucket checkpoint complete 2021-02-11T10:15:04.145 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:04.145 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:04.145 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:04.178 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:04.179 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:04.373 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 2021-02-11T10:15:04.405 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613038504.134025_941.1","last_update":"2021-02-11T10:15:04.134025Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:15:04.406 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:15:04.406 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:04.406 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:04.406 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:04.442 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:04.442 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:04.688 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038504.134025_941.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:15:04.134025Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:15:04.688 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:15:04.698 WARNING:tasks.rgw_multi.tests_ps:superfluous records are found 2021-02-11T10:15:04.723 INFO:tasks.rgw_multisite_tests:test notification status upon bucket deletion ... ok 2021-02-11T10:15:04.724 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:04.724 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:04.724 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:04.763 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:04.763 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:04.971 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 2021-02-11T10:15:05.010 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613038504.134025_941.1","last_update":"2021-02-11T10:15:04.134025Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:15:05.010 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T10:15:05.010 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:05.011 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:05.011 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:05.050 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:05.050 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:05.318 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038504.134025_941.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:15:04.134025Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:15:05.319 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T10:15:05.319 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T10:15:05.319 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:05.319 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:05.319 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:05.364 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:05.364 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:05.603 INFO:teuthology.orchestra.run.smithi094.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038504.134025_941.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:15:04.134025Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:15:05.604 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T10:15:05.604 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:15:05.604 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:05.604 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:05.604 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:05.649 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:05.649 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:05.897 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038504.134025_941.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:15:04.134025Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:15:05.898 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:15:05.898 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:05.898 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:05.898 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:05.939 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:05.939 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:06.154 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 2021-02-11T10:15:06.192 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613038504.134025_941.1","last_update":"2021-02-11T10:15:04.134025Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:15:06.192 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:15:06.193 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:06.193 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:06.193 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:06.234 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:06.234 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:06.527 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038504.134025_941.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:15:04.134025Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:15:06.528 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:15:06.576 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:06.576 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:06.576 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:06.611 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:06.611 INFO:teuthology.orchestra.run.smithi094.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:06.801 INFO:teuthology.orchestra.run.smithi094.stderr:No --period given, using current period=0d37c290-12ce-40b6-8dd3-c3f1d036f1a7 2021-02-11T10:15:06.841 INFO:teuthology.orchestra.run.smithi094.stdout:[{"marker":"1_1613038506.571501_948.1","last_update":"2021-02-11T10:15:06.571501Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T10:15:06.841 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T10:15:06.842 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:06.842 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T10:15:06.842 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T10:15:06.884 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setuser ceph since I am not root 2021-02-11T10:15:06.884 INFO:teuthology.orchestra.run.smithi098.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T10:15:07.274 INFO:teuthology.orchestra.run.smithi098.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d37c290-12ce-40b6-8dd3-c3f1d036f1a7","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613038506.571501_948.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T10:15:06.571501Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1613036864.081584_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T09:47:44.081584Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T10:15:07.274 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T10:15:07.283 INFO:tasks.rgw_multi.tests_ps:missing topic is expected 2021-02-11T10:15:07.393 INFO:tasks.rgw_multisite_tests:test creating a subscription when no topic info exists ... ok 2021-02-11T10:15:07.393 INFO:tasks.rgw_multisite_tests:test updating topic associated with a notification ... ok 2021-02-11T10:15:07.394 INFO:tasks.rgw_multisite_tests:test updating the topic of a notification ... ok 2021-02-11T10:15:07.394 INFO:tasks.rgw_multisite_tests:test notification creation with multiple topics ... ok 2021-02-11T10:15:07.394 INFO:tasks.rgw_multisite_tests: 2021-02-11T10:15:07.394 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2021-02-11T10:15:07.394 INFO:tasks.rgw_multisite_tests:Ran 55 tests in 112.261s 2021-02-11T10:15:07.394 INFO:tasks.rgw_multisite_tests: 2021-02-11T10:15:07.395 INFO:tasks.rgw_multisite_tests:OK 2021-02-11T10:15:07.395 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/run_tasks.py", line 94, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_ceph-c_5e5bb33c614118d912ca83c2e5f2be67b3acc04f/qa/tasks/rgw_multisite_tests.py", line 74, in begin raise RuntimeError(error_msg + 'test failures') RuntimeError: rgw multisite, test failures 2021-02-11T10:15:07.434 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=ff3178801d684083ad36779a51e12b6d Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/run_tasks.py", line 94, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_ceph-c_5e5bb33c614118d912ca83c2e5f2be67b3acc04f/qa/tasks/rgw_multisite_tests.py", line 74, in begin raise RuntimeError(error_msg + 'test failures') RuntimeError: rgw multisite, test failures 2021-02-11T10:15:07.435 DEBUG:teuthology.run_tasks:Unwinding manager rgw-multisite-tests 2021-02-11T10:15:07.442 DEBUG:teuthology.run_tasks:Unwinding manager rgw-multisite 2021-02-11T10:15:15.175 DEBUG:teuthology.run_tasks:Unwinding manager rgw 2021-02-11T10:15:29.112 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2021-02-11T10:15:29.112 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T10:15:29.132 INFO:tasks.rgw.c1.client.0.smithi094.stdout:2021-02-11T10:15:29.109+0000 7f74318fb700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 18532) UID: 0 2021-02-11T10:15:29.132 INFO:tasks.rgw.c1.client.0.smithi094.stdout:2021-02-11T10:15:29.109+0000 7f74494560c0 -1 shutting down 2021-02-11T10:15:29.736 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2021-02-11T10:15:30.237 INFO:tasks.rgw.c2.client.0.smithi098.stdout:2021-02-11T10:15:30.232+0000 7f2b0affd700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-5)ERROR: collect() returned error (ret=-5) 2021-02-11T10:15:34.487 INFO:tasks.rgw.c1.client.1.smithi094.stdout:2021-02-11T10:15:34.485+0000 7f8eeffff700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-5)ERROR: collect() returned error (ret=-5) 2021-02-11T10:15:35.213 INFO:tasks.rgw.c1.client.0:Stopped 2021-02-11T10:15:35.214 DEBUG:teuthology.orchestra.run.smithi094:> rm -f /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock 2021-02-11T10:15:35.220 DEBUG:teuthology.orchestra.run.smithi094:> rm -f /home/ubuntu/cephtest/vault-token 2021-02-11T10:15:35.267 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2021-02-11T10:15:35.268 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T10:15:35.311 INFO:tasks.rgw.c1.client.1.smithi094.stdout:2021-02-11T10:15:35.309+0000 7f8fc3fff700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 31141) UID: 0 2021-02-11T10:15:35.311 INFO:tasks.rgw.c1.client.1.smithi094.stdout:2021-02-11T10:15:35.309+0000 7f8fe7d630c0 -1 shutting down 2021-02-11T10:15:35.340 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~6s 2021-02-11T10:15:40.846 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2021-02-11T10:15:40.847 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2021-02-11T10:15:41.369 INFO:tasks.rgw.c1.client.1:Stopped 2021-02-11T10:15:41.369 DEBUG:teuthology.orchestra.run.smithi094:> rm -f /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock 2021-02-11T10:15:41.377 DEBUG:teuthology.orchestra.run.smithi094:> rm -f /home/ubuntu/cephtest/vault-token 2021-02-11T10:15:41.422 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2021-02-11T10:15:41.422 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T10:15:41.423 INFO:tasks.rgw.c2.client.0.smithi098.stdout:2021-02-11T10:15:41.420+0000 7f2be28b5700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 18503) UID: 0 2021-02-11T10:15:41.423 INFO:tasks.rgw.c2.client.0.smithi098.stdout:2021-02-11T10:15:41.420+0000 7f2bfa4100c0 -1 shutting down 2021-02-11T10:15:46.352 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2021-02-11T10:15:46.850 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2021-02-11T10:15:47.523 INFO:tasks.rgw.c2.client.0:Stopped 2021-02-11T10:15:47.524 DEBUG:teuthology.orchestra.run.smithi098:> rm -f /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock 2021-02-11T10:15:47.530 DEBUG:teuthology.orchestra.run.smithi098:> rm -f /home/ubuntu/cephtest/vault-token 2021-02-11T10:15:47.576 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2021-02-11T10:15:47.576 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T10:15:47.620 INFO:tasks.rgw.c2.client.1.smithi098.stdout:2021-02-11T10:15:47.616+0000 7fa83dd9f700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 19664) UID: 0 2021-02-11T10:15:47.620 INFO:tasks.rgw.c2.client.1.smithi098.stdout:2021-02-11T10:15:47.616+0000 7fa8558fa0c0 -1 shutting down 2021-02-11T10:15:52.354 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2021-02-11T10:15:57.234 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2021-02-11T10:15:57.236 INFO:tasks.rgw.c2.client.1:Stopped 2021-02-11T10:15:57.236 DEBUG:teuthology.orchestra.run.smithi098:> rm -f /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock 2021-02-11T10:15:57.243 DEBUG:teuthology.orchestra.run.smithi098:> rm -f /home/ubuntu/cephtest/vault-token 2021-02-11T10:15:57.289 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2021-02-11T10:15:57.297 INFO:tasks.ceph.ceph_manager.c2:Canceling any pending splits or merges... 2021-02-11T10:15:57.297 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2021-02-11T10:15:57.630 INFO:teuthology.orchestra.run.smithi098.stdout: 2021-02-11T10:15:57.630 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":70,"fsid":"0880a54d-a917-41e0-b7d6-67af274484d8","created":"2021-02-11T09:42:57.243402+0000","modified":"2021-02-11T09:58:50.516473+0000","last_up_change":"2021-02-11T09:43:04.825732+0000","last_in_change":"2021-02-11T09:43:04.825732+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-02-11T09:43:04.836347+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-02-11T09:43:08.767274+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"default.rgw.buckets.data","create_time":"2021-02-11T09:43:21.590647+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":4,"pool_name":"default.rgw.buckets.index","create_time":"2021-02-11T09:43:23.584400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":".rgw.root","create_time":"2021-02-11T09:43:44.294256+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.log","create_time":"2021-02-11T09:43:46.340432+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":7,"pool_name":"default.rgw.control","create_time":"2021-02-11T09:43:48.299485+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":8,"pool_name":"default.rgw.meta","create_time":"2021-02-11T09:43:50.411522+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":8},"application_metadata":{"rgw":{}}},{"pool":9,"pool_name":"test-zone2.rgw.buckets.data","create_time":"2021-02-11T09:44:15.570749+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"36","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":10,"pool_name":"test-zone2.rgw.log","create_time":"2021-02-11T09:44:17.552859+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":11,"pool_name":"test-zone2.rgw.control","create_time":"2021-02-11T09:44:19.348112+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":12,"pool_name":"test-zone2.rgw.meta","create_time":"2021-02-11T09:44:21.404346+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":8},"application_metadata":{"rgw":{}}},{"pool":13,"pool_name":"test-zone4.rgw.buckets.data","create_time":"2021-02-11T09:44:50.276481+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":14,"pool_name":"test-zone4.rgw.log","create_time":"2021-02-11T09:44:52.545035+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"48","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":15,"pool_name":"test-zone4.rgw.control","create_time":"2021-02-11T09:44:54.363094+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":16,"pool_name":"test-zone4.rgw.meta","create_time":"2021-02-11T09:44:56.423086+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":8},"application_metadata":{"rgw":{}}},{"pool":17,"pool_name":"test-zone2.rgw.buckets.index","create_time":"2021-02-11T09:47:23.383553+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":8},"application_metadata":{"rgw":{}}},{"pool":18,"pool_name":"test-zone4.rgw.buckets.index","create_time":"2021-02-11T09:47:23.394320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":8},"application_metadata":{"rgw":{}}},{"pool":19,"pool_name":"test-zone2.rgw.otp","create_time":"2021-02-11T09:51:06.240634+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":20,"pool_name":"remove.rgw.log","create_time":"2021-02-11T09:58:43.425512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":21,"pool_name":"remove.rgw.control","create_time":"2021-02-11T09:58:44.532137+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":22,"pool_name":"remove.rgw.meta","create_time":"2021-02-11T09:58:46.579554+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":8},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"73ab80cd-978c-4fda-9569-020ac3e0ac0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":14447},{"type":"v1","addr":"172.21.15.98:6808","nonce":14447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":14447},{"type":"v1","addr":"172.21.15.98:6813","nonce":14447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6821","nonce":14447},{"type":"v1","addr":"172.21.15.98:6823","nonce":14447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":14447},{"type":"v1","addr":"172.21.15.98:6818","nonce":14447}]},"public_addr":"172.21.15.98:6808/14447","cluster_addr":"172.21.15.98:6813/14447","heartbeat_back_addr":"172.21.15.98:6823/14447","heartbeat_front_addr":"172.21.15.98:6818/14447","state":["exists","up"]},{"osd":1,"uuid":"14a77ff0-2e1c-41c0-815a-8c0a87b02477","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":14448},{"type":"v1","addr":"172.21.15.98:6801","nonce":14448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":14448},{"type":"v1","addr":"172.21.15.98:6803","nonce":14448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6811","nonce":14448},{"type":"v1","addr":"172.21.15.98:6814","nonce":14448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":14448},{"type":"v1","addr":"172.21.15.98:6807","nonce":14448}]},"public_addr":"172.21.15.98:6801/14448","cluster_addr":"172.21.15.98:6803/14448","heartbeat_back_addr":"172.21.15.98:6814/14448","heartbeat_front_addr":"172.21.15.98:6807/14448","state":["exists","up"]},{"osd":2,"uuid":"2f71f5ca-5cbb-420e-ab80-7277fb21d403","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6805","nonce":14449},{"type":"v1","addr":"172.21.15.98:6809","nonce":14449}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":14449},{"type":"v1","addr":"172.21.15.98:6815","nonce":14449}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":14449},{"type":"v1","addr":"172.21.15.98:6822","nonce":14449}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6817","nonce":14449},{"type":"v1","addr":"172.21.15.98:6819","nonce":14449}]},"public_addr":"172.21.15.98:6809/14449","cluster_addr":"172.21.15.98:6815/14449","heartbeat_back_addr":"172.21.15.98:6822/14449","heartbeat_front_addr":"172.21.15.98:6819/14449","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T09:43:02.984492+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T09:43:02.789861+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T09:43:02.946888+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.1","mappings":[{"from":0,"to":1}]},{"pgid":"3.30","mappings":[{"from":0,"to":1}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.12","mappings":[{"from":1,"to":0}]},{"pgid":"4.21","mappings":[{"from":1,"to":2}]},{"pgid":"4.24","mappings":[{"from":1,"to":0}]},{"pgid":"4.3c","mappings":[{"from":1,"to":2}]},{"pgid":"10.12","mappings":[{"from":1,"to":0}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-02-11T10:15:57.649 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --cluster c2 config set global mon_health_to_clog false 2021-02-11T10:15:57.973 INFO:teuthology.misc:Shutting down mds daemons... 2021-02-11T10:15:57.973 INFO:teuthology.misc:Shutting down osd daemons... 2021-02-11T10:15:57.973 DEBUG:tasks.ceph.c2.osd.0:waiting for process to exit 2021-02-11T10:15:57.973 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T10:15:58.044 INFO:tasks.ceph.c2.osd.0:Stopped 2021-02-11T10:15:58.045 DEBUG:tasks.ceph.c2.osd.1:waiting for process to exit 2021-02-11T10:15:58.045 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T10:15:58.122 INFO:tasks.ceph.c2.osd.1:Stopped 2021-02-11T10:15:58.123 DEBUG:tasks.ceph.c2.osd.2:waiting for process to exit 2021-02-11T10:15:58.123 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T10:15:58.216 INFO:tasks.ceph.c2.osd.2:Stopped 2021-02-11T10:15:58.216 INFO:teuthology.misc:Shutting down mgr daemons... 2021-02-11T10:15:58.216 DEBUG:tasks.ceph.c2.mgr.x:waiting for process to exit 2021-02-11T10:15:58.217 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T10:15:58.241 INFO:tasks.ceph.c2.mgr.x:Stopped 2021-02-11T10:15:58.242 INFO:teuthology.misc:Shutting down mon daemons... 2021-02-11T10:15:58.242 DEBUG:tasks.ceph.c2.mon.a:waiting for process to exit 2021-02-11T10:15:58.242 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T10:15:58.307 INFO:tasks.ceph.c2.mon.a:Stopped 2021-02-11T10:15:58.307 INFO:tasks.ceph:Checking cluster log for badness... 2021-02-11T10:15:58.307 DEBUG:teuthology.orchestra.run.smithi098:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | head -n 1 2021-02-11T10:15:58.365 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-0 on ubuntu@smithi098.front.sepia.ceph.com 2021-02-11T10:15:58.365 DEBUG:teuthology.orchestra.run.smithi098:> sync && sudo umount -f /var/lib/ceph/osd/c2-0 2021-02-11T10:15:59.062 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-1 on ubuntu@smithi098.front.sepia.ceph.com 2021-02-11T10:15:59.063 DEBUG:teuthology.orchestra.run.smithi098:> sync && sudo umount -f /var/lib/ceph/osd/c2-1 2021-02-11T10:15:59.222 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-2 on ubuntu@smithi098.front.sepia.ceph.com 2021-02-11T10:15:59.222 DEBUG:teuthology.orchestra.run.smithi098:> sync && sudo umount -f /var/lib/ceph/osd/c2-2 2021-02-11T10:15:59.370 INFO:tasks.ceph:Archiving mon data... 2021-02-11T10:15:59.372 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/lib/ceph/mon/c2-a to /home/teuthworker/archive/aemerson-2021-02-11_09:24:13-rgw-wip-blurpletastic-distro-basic-smithi/5874501/data/c2.mon.a.tgz 2021-02-11T10:15:59.372 DEBUG:teuthology.orchestra.run.smithi098:> mktemp 2021-02-11T10:15:59.378 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/tmp.ZWXo4tyRz3 2021-02-11T10:15:59.378 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar cz -f - -C /var/lib/ceph/mon/c2-a -- . > /tmp/tmp.ZWXo4tyRz3 2021-02-11T10:15:59.804 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0666 /tmp/tmp.ZWXo4tyRz3 2021-02-11T10:15:59.867 DEBUG:teuthology.orchestra.remote:smithi098:/tmp/tmp.ZWXo4tyRz3 is 2MB 2021-02-11T10:15:59.947 DEBUG:teuthology.orchestra.run.smithi098:> rm -fr /tmp/tmp.ZWXo4tyRz3 2021-02-11T10:15:59.952 INFO:tasks.ceph:Cleaning ceph cluster... 2021-02-11T10:15:59.952 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -rf -- /etc/ceph/c2.conf /etc/ceph/c2.keyring /home/ubuntu/cephtest/c2.data /home/ubuntu/cephtest/c2.monmap /home/ubuntu/cephtest/../*.pid 2021-02-11T10:15:59.954 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -rf -- /etc/ceph/c2.conf /etc/ceph/c2.keyring /home/ubuntu/cephtest/c2.data /home/ubuntu/cephtest/c2.monmap /home/ubuntu/cephtest/../*.pid 2021-02-11T10:16:00.010 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2021-02-11T10:16:03.499 INFO:tasks.ceph.ceph_manager.c1:Canceling any pending splits or merges... 2021-02-11T10:16:03.500 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2021-02-11T10:16:03.500 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 has been restored 2021-02-11T10:16:03.804 INFO:teuthology.orchestra.run.smithi094.stdout: 2021-02-11T10:16:03.804 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":66,"fsid":"67a12f0c-a3f1-4f95-b0bb-3bdc01f1c8cf","created":"2021-02-11T09:42:25.208182+0000","modified":"2021-02-11T10:04:33.534461+0000","last_up_change":"2021-02-11T09:42:32.923194+0000","last_in_change":"2021-02-11T09:42:32.923194+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-02-11T09:42:32.937534+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-02-11T09:42:38.378396+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"default.rgw.buckets.data","create_time":"2021-02-11T09:43:15.872933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":4,"pool_name":"default.rgw.buckets.index","create_time":"2021-02-11T09:43:17.581114+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":".rgw.root","create_time":"2021-02-11T09:43:27.346780+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.log","create_time":"2021-02-11T09:43:29.294251+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":7,"pool_name":"default.rgw.control","create_time":"2021-02-11T09:43:31.263073+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":8,"pool_name":"default.rgw.meta","create_time":"2021-02-11T09:43:33.313629+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":8},"application_metadata":{"rgw":{}}},{"pool":9,"pool_name":"test-zone1.rgw.buckets.data","create_time":"2021-02-11T09:43:57.530392+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":10,"pool_name":"test-zone1.rgw.log","create_time":"2021-02-11T09:43:59.488928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"36","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":11,"pool_name":"test-zone1.rgw.control","create_time":"2021-02-11T09:44:01.281444+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":12,"pool_name":"test-zone1.rgw.meta","create_time":"2021-02-11T09:44:03.331550+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":8},"application_metadata":{"rgw":{}}},{"pool":13,"pool_name":"test-zone1.rgw.otp","create_time":"2021-02-11T09:44:32.333442+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":14,"pool_name":"test-zone3.rgw.buckets.data","create_time":"2021-02-11T09:44:33.243710+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":15,"pool_name":"test-zone3.rgw.log","create_time":"2021-02-11T09:44:34.477158+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"48","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":16,"pool_name":"test-zone3.rgw.control","create_time":"2021-02-11T09:44:36.337071+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":17,"pool_name":"test-zone3.rgw.meta","create_time":"2021-02-11T09:44:38.369804+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":8},"application_metadata":{"rgw":{}}},{"pool":18,"pool_name":"test-zone1.rgw.buckets.index","create_time":"2021-02-11T09:47:19.461785+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":8},"application_metadata":{"rgw":{}}},{"pool":19,"pool_name":"test-zone3.rgw.buckets.index","create_time":"2021-02-11T09:47:23.387615+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":8},"application_metadata":{"rgw":{}}},{"pool":20,"pool_name":"test-zone1.rgw.buckets.non-ec","create_time":"2021-02-11T10:04:30.828733+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"5ad1fdb3-bb5e-43f4-92ee-e522fd99ffb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":14390},{"type":"v1","addr":"172.21.15.94:6801","nonce":14390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":14390},{"type":"v1","addr":"172.21.15.94:6803","nonce":14390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6809","nonce":14390},{"type":"v1","addr":"172.21.15.94:6811","nonce":14390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6805","nonce":14390},{"type":"v1","addr":"172.21.15.94:6807","nonce":14390}]},"public_addr":"172.21.15.94:6801/14390","cluster_addr":"172.21.15.94:6803/14390","heartbeat_back_addr":"172.21.15.94:6811/14390","heartbeat_front_addr":"172.21.15.94:6807/14390","state":["exists","up"]},{"osd":1,"uuid":"f51bf15f-b1fb-47c2-8586-8a36cfcf8ab6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":14389},{"type":"v1","addr":"172.21.15.94:6806","nonce":14389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":14389},{"type":"v1","addr":"172.21.15.94:6810","nonce":14389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":14389},{"type":"v1","addr":"172.21.15.94:6815","nonce":14389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":14389},{"type":"v1","addr":"172.21.15.94:6813","nonce":14389}]},"public_addr":"172.21.15.94:6806/14389","cluster_addr":"172.21.15.94:6810/14389","heartbeat_back_addr":"172.21.15.94:6815/14389","heartbeat_front_addr":"172.21.15.94:6813/14389","state":["exists","up"]},{"osd":2,"uuid":"fa9f973f-5860-4200-bbfd-7053478e8a62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":14391},{"type":"v1","addr":"172.21.15.94:6817","nonce":14391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":14391},{"type":"v1","addr":"172.21.15.94:6819","nonce":14391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":14391},{"type":"v1","addr":"172.21.15.94:6823","nonce":14391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":14391},{"type":"v1","addr":"172.21.15.94:6821","nonce":14391}]},"public_addr":"172.21.15.94:6817/14391","cluster_addr":"172.21.15.94:6819/14391","heartbeat_back_addr":"172.21.15.94:6823/14391","heartbeat_front_addr":"172.21.15.94:6821/14391","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T09:42:30.925557+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T09:42:30.834864+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T09:42:30.923398+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.12","mappings":[{"from":0,"to":2}]},{"pgid":"3.18","mappings":[{"from":0,"to":1}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.15","mappings":[{"from":1,"to":2}]},{"pgid":"4.27","mappings":[{"from":1,"to":0}]},{"pgid":"4.2c","mappings":[{"from":1,"to":0}]},{"pgid":"4.30","mappings":[{"from":1,"to":2}]},{"pgid":"4.38","mappings":[{"from":1,"to":2}]},{"pgid":"4.3f","mappings":[{"from":1,"to":0}]},{"pgid":"10.1a","mappings":[{"from":1,"to":0}]},{"pgid":"14.11","mappings":[{"from":1,"to":0}]},{"pgid":"14.20","mappings":[{"from":1,"to":2}]},{"pgid":"14.2e","mappings":[{"from":1,"to":2}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-02-11T10:16:03.819 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph --cluster c1 config set global mon_health_to_clog false 2021-02-11T10:16:04.152 INFO:teuthology.misc:Shutting down mds daemons... 2021-02-11T10:16:04.152 INFO:teuthology.misc:Shutting down osd daemons... 2021-02-11T10:16:04.152 DEBUG:tasks.ceph.c1.osd.0:waiting for process to exit 2021-02-11T10:16:04.152 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T10:16:04.228 INFO:tasks.ceph.c1.osd.0:Stopped 2021-02-11T10:16:04.229 DEBUG:tasks.ceph.c1.osd.1:waiting for process to exit 2021-02-11T10:16:04.229 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T10:16:04.316 INFO:tasks.ceph.c1.osd.1:Stopped 2021-02-11T10:16:04.317 DEBUG:tasks.ceph.c1.osd.2:waiting for process to exit 2021-02-11T10:16:04.317 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T10:16:04.401 INFO:tasks.ceph.c1.osd.2:Stopped 2021-02-11T10:16:04.401 INFO:teuthology.misc:Shutting down mgr daemons... 2021-02-11T10:16:04.401 DEBUG:tasks.ceph.c1.mgr.x:waiting for process to exit 2021-02-11T10:16:04.401 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T10:16:04.424 INFO:tasks.ceph.c1.mgr.x:Stopped 2021-02-11T10:16:04.424 INFO:teuthology.misc:Shutting down mon daemons... 2021-02-11T10:16:04.425 DEBUG:tasks.ceph.c1.mon.a:waiting for process to exit 2021-02-11T10:16:04.425 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T10:16:04.492 INFO:tasks.ceph.c1.mon.a:Stopped 2021-02-11T10:16:04.492 INFO:tasks.ceph:Checking cluster log for badness... 2021-02-11T10:16:04.492 DEBUG:teuthology.orchestra.run.smithi094:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | head -n 1 2021-02-11T10:16:04.550 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-0 on ubuntu@smithi094.front.sepia.ceph.com 2021-02-11T10:16:04.550 DEBUG:teuthology.orchestra.run.smithi094:> sync && sudo umount -f /var/lib/ceph/osd/c1-0 2021-02-11T10:16:04.973 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-1 on ubuntu@smithi094.front.sepia.ceph.com 2021-02-11T10:16:04.973 DEBUG:teuthology.orchestra.run.smithi094:> sync && sudo umount -f /var/lib/ceph/osd/c1-1 2021-02-11T10:16:05.228 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-2 on ubuntu@smithi094.front.sepia.ceph.com 2021-02-11T10:16:05.229 DEBUG:teuthology.orchestra.run.smithi094:> sync && sudo umount -f /var/lib/ceph/osd/c1-2 2021-02-11T10:16:05.388 INFO:tasks.ceph:Archiving mon data... 2021-02-11T10:16:05.389 DEBUG:teuthology.misc:Transferring archived files from smithi094:/var/lib/ceph/mon/c1-a to /home/teuthworker/archive/aemerson-2021-02-11_09:24:13-rgw-wip-blurpletastic-distro-basic-smithi/5874501/data/c1.mon.a.tgz 2021-02-11T10:16:05.389 DEBUG:teuthology.orchestra.run.smithi094:> mktemp 2021-02-11T10:16:05.394 INFO:teuthology.orchestra.run.smithi094.stdout:/tmp/tmp.Uea5XNATm5 2021-02-11T10:16:05.395 DEBUG:teuthology.orchestra.run.smithi094:> sudo tar cz -f - -C /var/lib/ceph/mon/c1-a -- . > /tmp/tmp.Uea5XNATm5 2021-02-11T10:16:05.756 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 0666 /tmp/tmp.Uea5XNATm5 2021-02-11T10:16:05.818 DEBUG:teuthology.orchestra.remote:smithi094:/tmp/tmp.Uea5XNATm5 is 1MB 2021-02-11T10:16:05.889 DEBUG:teuthology.orchestra.run.smithi094:> rm -fr /tmp/tmp.Uea5XNATm5 2021-02-11T10:16:05.896 INFO:tasks.ceph:Cleaning ceph cluster... 2021-02-11T10:16:05.896 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -rf -- /etc/ceph/c1.conf /etc/ceph/c1.keyring /home/ubuntu/cephtest/c1.data /home/ubuntu/cephtest/c1.monmap /home/ubuntu/cephtest/../*.pid 2021-02-11T10:16:05.939 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -rf -- /etc/ceph/c1.conf /etc/ceph/c1.keyring /home/ubuntu/cephtest/c1.data /home/ubuntu/cephtest/c1.monmap /home/ubuntu/cephtest/../*.pid 2021-02-11T10:16:05.956 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2021-02-11T10:16:05.956 DEBUG:teuthology.orchestra.run.smithi094:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2021-02-11T10:16:05.995 DEBUG:teuthology.orchestra.run.smithi098:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2021-02-11T10:16:06.016 INFO:teuthology.orchestra.run.smithi094.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2021-02-11T10:16:06.028 INFO:teuthology.orchestra.run.smithi098.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2021-02-11T10:16:06.036 INFO:tasks.ceph:Archiving crash dumps... 2021-02-11T10:16:06.039 DEBUG:teuthology.misc:Transferring archived files from smithi094:/var/lib/ceph/crash to /home/teuthworker/archive/aemerson-2021-02-11_09:24:13-rgw-wip-blurpletastic-distro-basic-smithi/5874501/remote/smithi094/crash 2021-02-11T10:16:06.040 DEBUG:teuthology.orchestra.run.smithi094:> sudo tar cz -f - -C /var/lib/ceph/crash -- . 2021-02-11T10:16:06.083 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/lib/ceph/crash to /home/teuthworker/archive/aemerson-2021-02-11_09:24:13-rgw-wip-blurpletastic-distro-basic-smithi/5874501/remote/smithi098/crash 2021-02-11T10:16:06.084 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar cz -f - -C /var/lib/ceph/crash -- . 2021-02-11T10:16:06.101 INFO:tasks.ceph:Compressing logs... 2021-02-11T10:16:06.101 DEBUG:teuthology.orchestra.run.smithi094:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2021-02-11T10:16:06.127 DEBUG:teuthology.orchestra.run.smithi098:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2021-02-11T10:17:31.508 INFO:tasks.ceph:Archiving logs... 2021-02-11T10:17:31.508 DEBUG:teuthology.misc:Transferring archived files from smithi094:/var/log/ceph to /home/teuthworker/archive/aemerson-2021-02-11_09:24:13-rgw-wip-blurpletastic-distro-basic-smithi/5874501/remote/smithi094/log 2021-02-11T10:17:31.509 DEBUG:teuthology.orchestra.run.smithi094:> sudo tar cz -f - -C /var/log/ceph -- . 2021-02-11T10:17:47.361 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/log/ceph to /home/teuthworker/archive/aemerson-2021-02-11_09:24:13-rgw-wip-blurpletastic-distro-basic-smithi/5874501/remote/smithi098/log 2021-02-11T10:17:47.362 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar cz -f - -C /var/log/ceph -- . 2021-02-11T10:18:01.489 DEBUG:teuthology.run_tasks:Unwinding manager install 2021-02-11T10:18:01.505 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits... 2021-02-11T10:18:01.505 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2021-02-11T10:18:01.508 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2021-02-11T10:18:01.555 DEBUG:teuthology.run_tasks:Unwinding manager clock 2021-02-11T10:18:05.057 INFO:teuthology.task.clock:Checking final clock skew... 2021-02-11T10:18:10.428 DEBUG:teuthology.orchestra.run.smithi094:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-02-11T10:18:10.451 INFO:teuthology.orchestra.run.smithi094.stdout: remote refid st t when poll reach delay offset jitter 2021-02-11T10:18:10.451 INFO:teuthology.orchestra.run.smithi094.stdout:============================================================================== 2021-02-11T10:18:10.452 INFO:teuthology.orchestra.run.smithi094.stdout:-hv01.front.sepi 216.240.36.24 3 u 115 128 377 0.060 -0.567 0.108 2021-02-11T10:18:10.452 INFO:teuthology.orchestra.run.smithi094.stdout:+hv02.front.sepi 163.237.218.18 2 u 120 128 377 0.080 0.882 0.167 2021-02-11T10:18:10.452 INFO:teuthology.orchestra.run.smithi094.stdout:+hv03.front.sepi 74.6.168.72 3 u 125 128 377 0.085 1.751 0.091 2021-02-11T10:18:10.452 INFO:teuthology.orchestra.run.smithi094.stdout:*hv04.front.sepi 144.172.126.201 3 u 46 128 377 0.072 0.673 0.073 2021-02-11T10:18:10.453 DEBUG:teuthology.orchestra.run.smithi098:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-02-11T10:18:10.474 INFO:teuthology.orchestra.run.smithi098.stdout: remote refid st t when poll reach delay offset jitter 2021-02-11T10:18:10.474 INFO:teuthology.orchestra.run.smithi098.stdout:============================================================================== 2021-02-11T10:18:10.475 INFO:teuthology.orchestra.run.smithi098.stdout:-hv01.front.sepi 216.240.36.24 3 u 116 128 377 0.093 -0.100 1.310 2021-02-11T10:18:10.475 INFO:teuthology.orchestra.run.smithi098.stdout:+hv02.front.sepi 163.237.218.18 2 u 37 128 377 0.093 2.914 1.255 2021-02-11T10:18:10.475 INFO:teuthology.orchestra.run.smithi098.stdout:+hv03.front.sepi 74.6.168.72 3 u 112 128 377 0.076 4.011 1.190 2021-02-11T10:18:10.475 INFO:teuthology.orchestra.run.smithi098.stdout:*hv04.front.sepi 144.172.126.201 3 u 46 128 377 0.068 1.432 1.062 2021-02-11T10:18:10.476 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2021-02-11T10:18:10.509 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2021-02-11T10:18:10.509 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2021-02-11T10:18:19.090 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2021-02-11T10:18:36.767 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2021-02-11T10:18:36.797 INFO:teuthology.task.internal:Duration was 2469.009634 seconds 2021-02-11T10:18:36.799 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2021-02-11T10:18:36.824 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2021-02-11T10:18:36.825 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2021-02-11T10:18:36.828 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2021-02-11T10:18:36.879 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2021-02-11T10:18:36.879 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi094.front.sepia.ceph.com 2021-02-11T10:18:36.879 DEBUG:teuthology.orchestra.run.smithi094:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/*.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2021-02-11T10:18:36.889 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi098.front.sepia.ceph.com 2021-02-11T10:18:36.889 DEBUG:teuthology.orchestra.run.smithi098:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/*.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2021-02-11T10:18:36.900 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2021-02-11T10:18:36.900 DEBUG:teuthology.orchestra.run.smithi094:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2021-02-11T10:18:36.932 DEBUG:teuthology.orchestra.run.smithi098:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2021-02-11T10:18:36.971 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2021-02-11T10:18:36.983 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2021-02-11T10:18:36.983 DEBUG:teuthology.orchestra.run.smithi094:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2021-02-11T10:18:37.008 DEBUG:teuthology.orchestra.run.smithi098:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2021-02-11T10:18:37.030 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2021-02-11T10:18:37.114 DEBUG:teuthology.orchestra.run.smithi094:> sudo sysctl -w kernel.core_pattern=core && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2021-02-11T10:18:37.116 DEBUG:teuthology.orchestra.run.smithi098:> sudo sysctl -w kernel.core_pattern=core && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2021-02-11T10:18:37.127 INFO:teuthology.orchestra.run.smithi094.stdout:kernel.core_pattern = core 2021-02-11T10:18:37.130 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern = core 2021-02-11T10:18:37.229 DEBUG:teuthology.orchestra.run.smithi094:> test -e /home/ubuntu/cephtest/archive/coredump 2021-02-11T10:18:37.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-11T10:18:37.275 DEBUG:teuthology.orchestra.run.smithi098:> test -e /home/ubuntu/cephtest/archive/coredump 2021-02-11T10:18:37.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-11T10:18:37.280 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2021-02-11T10:18:37.329 INFO:teuthology.task.internal:Transferring archived files... 2021-02-11T10:18:37.330 DEBUG:teuthology.misc:Transferring archived files from smithi094:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/aemerson-2021-02-11_09:24:13-rgw-wip-blurpletastic-distro-basic-smithi/5874501/remote/smithi094 2021-02-11T10:18:37.330 DEBUG:teuthology.orchestra.run.smithi094:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . 2021-02-11T10:18:37.354 DEBUG:teuthology.misc:Transferring archived files from smithi098:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/aemerson-2021-02-11_09:24:13-rgw-wip-blurpletastic-distro-basic-smithi/5874501/remote/smithi098 2021-02-11T10:18:37.354 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . 2021-02-11T10:18:37.377 INFO:teuthology.task.internal:Removing archive directory... 2021-02-11T10:18:37.377 DEBUG:teuthology.orchestra.run.smithi094:> rm -rf -- /home/ubuntu/cephtest/archive 2021-02-11T10:18:37.395 DEBUG:teuthology.orchestra.run.smithi098:> rm -rf -- /home/ubuntu/cephtest/archive 2021-02-11T10:18:37.425 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2021-02-11T10:18:37.438 INFO:teuthology.task.internal:Not uploading archives. 2021-02-11T10:18:37.438 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2021-02-11T10:18:37.474 INFO:teuthology.task.internal:Tidying up after the test... 2021-02-11T10:18:37.475 DEBUG:teuthology.orchestra.run.smithi094:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2021-02-11T10:18:37.477 DEBUG:teuthology.orchestra.run.smithi098:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2021-02-11T10:18:37.482 INFO:teuthology.orchestra.run.smithi094.stdout: 524339 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 11 10:18 /home/ubuntu/cephtest 2021-02-11T10:18:37.485 INFO:teuthology.orchestra.run.smithi098.stdout: 524339 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 11 10:18 /home/ubuntu/cephtest 2021-02-11T10:18:37.487 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2021-02-11T10:18:37.650 INFO:teuthology.nuke:Checking targets against current locks 2021-02-11T10:18:37.738 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/bin/teuthology", line 33, in sys.exit(load_entry_point('teuthology', 'console_scripts', 'teuthology')()) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/scripts/run.py", line 38, in main teuthology.run.main(args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/run.py", line 412, in main report_outcome(config, archive, fake_ctx.summary, fake_ctx) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/run.py", line 269, in report_outcome nuke(fake_ctx, fake_ctx.lock) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/nuke/__init__.py", line 266, in nuke for unnuked in p: File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/nuke/__init__.py", line 280, in nuke_one keep_logs = ctx.keep_logs File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/config.py", line 255, in __getattr__ raise AttributeError(name) AttributeError: keep_logs